How to run your first website crawl

Running your first website crawl doesn't require a manual or a certification. You need a URL, a few minutes, and a clear idea of what you want from the results. Here's a workflow that works whether the site has 50 pages or 50,000.

1. Pick your starting URL and scope

Enter the full URL of the site (e.g. https://example.com). The crawler will start there and follow links. Most tools let you restrict scope—for example, same domain only or same subdomain. For a typical SEO audit, same domain is enough. Restricting scope keeps the crawl faster and focused on the site you care about.

2. Run the crawl

Hit start and let the crawler run. It will discover pages, run checks, and flag issues. How long it takes depends on site size and your settings (e.g. request delay). You don't need to watch it; when it's done, you'll get a list of URLs and a list of issues tied to those URLs.

3. Don't fix everything at once

A full crawl often surfaces a lot of issues. The mistake is trying to fix all of them in one go. Use the Fix Plan (or whatever your tool uses for prioritization). It should rank issues by impact and effort so you see a Top 10 and Quick Wins. Fix those first, then reassess. That's how you get visible results without burning out.

4. Export a proof report

If you're working for a client or an internal stakeholder, export a report. A good proof report shows what was crawled, what was found, and what you recommend fixing first. That way the value of the audit is clear and you have something to reference later. Frog Radar lets you export client-ready PDFs so you can hand off the crawl results and the Fix Plan in one go.

Next step

Download Frog Radar for Windows or macOS, enter a URL, and run your first crawl. Then open the Fix Plan and export a proof report. Once you've done that, you've got a repeatable process for any site.