Whole-site crawl

The crawl reads every page it can find on a site and scores each one the way the homepage scan does, then rolls the results up into whole-site averages, an issue list and a per-page table.

Plan: Starter (25 pages) and Pro / Enterprise (the full site, up to 5,000 pages). Free has no crawl; the endpoints answer 402.

Where

• Web: SEO, the domain, Crawl every page; Pages for the results.
• iOS: SEO & accessibility, Pages, or Crawl every page on the Website module. The button reads "Crawling 41/152" with a progress bar.
• API/MCP: POST /api/site-audit/start, GET /api/site-audit/status, GET /api/site-audit/results; tools start_site_audit, get_site_audit_status, get_site_audit_results. The older deep-SEO job endpoints (/api/deep-seo/jobs, fetch_sitemap, start_deep_seo_job) crawl a list of pages you choose.

How pages are found

1. The sitemap (and sitemap index) is read.
2. The site's own links are followed from the homepage and every discovered page.
3. The two lists are merged. When pages came from links rather than the sitemap, the crawl says so.

Duplicates are collapsed: /about and /about/ are one page, and one page linked with fifty different query strings is one page. On a real 300-page site that once doubled the crawl, and real pages fell off the end of the cap.

The cap

A Starter crawl stops at 25 pages and the report says "25 of 152 pages checked", which is the number that tells you how much of the site is unmeasured. Pro and Enterprise crawl to the full site up to 5,000 pages. When your page list is trimmed the response says so.

What each page gets

Title and description with lengths, heading outline, word count, images and alt text, links in and out, canonical, indexability, structured data, and the same accessibility and mobile checks as the homepage. Each page has its own score and a defects list.

Running in the background

A crawl is asynchronous. Start it, poll the status, then read the results. In the app the page checking runs at a lower priority than anything you touch. Cancelling keeps every page already scored.

Rollups

The SEO section shows whole-site averages once a crawl has run, the issues grouped by rule with the affected pages and their word counts, and the search intent report, which is computed from the newest crawl.

Common questions

The crawl found pages my sitemap does not list. Link discovery found them. Either add them to the sitemap or, if they should not be indexed, mark them noindex.

Some pages show as failed. A page that returned an error or timed out is counted as failed, not as a zero score.

How often should I crawl? After a content change, or monthly. Pro and Enterprise accounts get an automatic monthly SEO scan; see Reports.