Website and SEO

Whole-site crawl

How the crawl finds pages (sitemap plus links), what it checks on each, the 25-page Starter cap, the 300-page site audit and the 5,000-page deep crawl, progress, cancelling and per-page.

Updated · 4 min read

The crawl reads every page it can find on a site and scores each one the way the homepage scan does, then rolls the results up into whole-site averages, an issue list and a per-page table.

Plan: Starter (25 pages) and Pro / Enterprise. On Pro and Enterprise the web's Audit every page (the site audit, start_site_audit) covers up to 300 pages; the app's crawl (the deep-SEO crawl, /api/deep-seo/jobs) covers up to 5,000. Free has no crawl; the endpoints answer 402.

Where

  • Web: SEO in the rail, open the domain, Whole-site audit, Audit every page; or a domain's SEO & search tab, Crawl site, with the results under Pages.
  • iOS: SEO & accessibility, Pages, or Crawl every page on the Website module. The button reads "Crawling 41/152" with a progress bar. The app runs the deep-SEO crawl, not the site audit.
  • API/MCP: POST /api/site-audit/start, GET /api/site-audit/status, GET /api/site-audit/results; tools start_site_audit, get_site_audit_status, get_site_audit_results. The older deep-SEO job endpoints (/api/deep-seo/jobs, fetch_sitemap, start_deep_seo_job) crawl a list of pages you choose, up to 5,000 on Pro and Enterprise.

How pages are found

  1. The sitemap (and sitemap index) is read.
  2. The site's own links are followed from the homepage and every discovered page.
  3. The two lists are merged. When pages came from links rather than the sitemap, the crawl says so.

Duplicates are collapsed: /about and /about/ are one page, and one page linked with fifty different query strings counts as at most three pages (enough to keep real pagination such as ?page=2). On a real 300-page site that once doubled the crawl, and real pages fell off the end of the cap.

The cap

A Starter crawl stops at 25 pages and the report says "25 of 152 pages checked", which is the number that tells you how much of the site is unmeasured. On Pro and Enterprise the site audit (web Audit every page, start_site_audit) stops at 300 pages, the most one run can fetch; the deep-SEO crawl the app uses goes up to 5,000 pages. When your page list is trimmed the response says so.

What each page gets

Title and description with lengths, heading outline, word count, images and alt text, links in and out, canonical, indexability, structured data, and the same accessibility and mobile checks as the homepage. Each page has its own score and a defects list. Every defect carries the same fix as the homepage scan: what is on the page now, what to change, a draft where the page gives one, and where in your site builder to do it. A page over 256KB of HTML is read up to that point rather than skipped, and a heading or structured-data verdict is not given for a page that was cut off.

Templated pages

When the deep-SEO crawl finishes it compares the pages with each other, which no per-page check can do. Pages whose main text is mostly the same (an estimated 80% or more of their five-word runs in common, once each page's own name from its URL is set aside) are reported in groups of three or more as Templated Pages, on every page in the group. That is the city page copied for every town with the name swapped, which Google's spam policies call scaled content or doorway pages. Product pages and pages that are noindex or point their canonical elsewhere are left out, and pages with too little text are not compared. Pages rewritten for each town, with their own sentences, are not flagged; the check would rather miss a pattern than accuse a site of one. What to do about a group is in AI-written content and location pages.

Running in the background

A crawl is asynchronous. Start it, poll the status, then read the results. In the app the page checking runs at a lower priority than anything you touch. Cancelling keeps every page already scored.

Rollups

The SEO & search tab shows whole-site averages once a crawl has run, the issues grouped by rule with the affected pages and their word counts, and the search intent report, which is computed from the newest crawl.

Common questions

The crawl found pages my sitemap does not list. Link discovery found them. Either add them to the sitemap or, if they should not be indexed, mark them noindex.

Some pages show as failed. A page that returned an error or timed out is counted as failed, not as a zero score.

How often should I crawl? After a content change, or monthly. Pro and Enterprise get an automatic homepage SEO scan every day and Starter every week; see Reports.

Keep reading

Related articles

  1. The site check: speed, mobile, SEO and security in a real browserHow the site check loads your pages as a phone on 4G, what the six sections measure, how each finding tells you the cause and the fix, and when checks run on their own.Website and SEO ·Updated
  2. IndexNow: tell search engines a page changedTurning IndexNow on for a domain, the key file your site has to serve, which engines get notified, sending pages by hand or from the sitemap, and the record of every URL sent and what each engine answered.Website and SEO ·Updated
  3. The five tabsMonitoring, Alerts, Scan, Exposure and Account: what each tab answers, what is on it, and the deep links and Home Screen quick actions that open each.iOS app ·Updated

Still stuck?

Ask the people who run the scanner.

Send the domain and what you expected to see. We look at the same scan you are looking at and write back with what it means and what to change.

Contact supportCall (330) 305-2750