Quick Start Guide

Get your first website crawl running in under 5 minutes. Follow these simple steps to visualize your internal link structure and uncover SEO insights.

1. Create an Account

Sign up at the login page using your email via Supabase Auth. The free plan includes up to 50 pages per crawl and 5 crawls per month — no credit card required.

2. Enter Your URL

Once logged in, you'll see the dashboard with a URL input in the top bar. Enter the root URL of the website you want to analyze (e.g., https://example.com) and click "Crawl".

The system creates a crawl job, publishes it to the job queue (Upstash QStash), and redirects you to the job detail page.

💡Always use the root domain as the starting URL. The crawler uses BFS and will automatically discover all reachable internal pages from there.

3. Monitor Progress

On the job detail page, you'll see a live progress indicator showing pages crawled out of the maximum limit. The crawler processes up to 5 pages concurrently with a 200ms delay between batches to be respectful of target servers.

4. Explore Results

Once the crawl completes, an interactive graph visualization loads with your site's structure. Switch between layout modes:

Rings: Concentric circles by depth — see click distance from homepage.
Tree: Hierarchical tree showing parent-child BFS relationships.
Force: Physics simulation revealing natural content clusters.
Silos: Column layout grouping pages by URL path structure.

Use the metrics sidebar to review your architecture score, depth distribution, orphan pages, and link statistics.

5. Analyze Topics

After the crawl completes, topic analysis is automatically triggered. The system extracts text content from each page (using smart noise removal), generates embeddings, and groups pages into topical clusters. Browse the results in the Topic Browser to see how your content is organized.

Configure Settings

Visit the Settings page to configure:

Global exclude patterns: Comma-separated URL patterns to skip during crawling (e.g., /admin/*, /wp-json/*).

Exclude patterns support wildcards and are applied to every crawl you run.

Tips for Better Results

Ensure your website is publicly accessible before crawling.
Check that your robots.txt doesn't block the pages you want to analyze.
The crawler respects nofollow link attributes — these links are skipped.
Asset URLs (images, PDFs, CSS, JS) are automatically skipped during crawling.
For large sites, consider upgrading to a higher plan for increased page limits.
Use the History page to compare crawl results over time.