Checks / AI crawler access
Pages return an error status
Pages that answer 4xx or 5xx (or do not answer at all) cannot be read, indexed or cited.
page_status · AI crawler access · Error
What triggers it
A page we crawled answers with a status of 400 or higher, or does not answer.
What to change
Fix or redirect the page; remove links that point to it.
How to confirm the fix
- Open the URL in a private window: it should load without an error page
- Press 'Re-check 1 page' on the page: the status shows 200
How common it is
We have not measured how common this is on a sample of sites. Counts appear here when a study covers it.
About this check
| Check ID | page_status |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | No, the fix is described in words |
Related checks
- Your firewall or CDN refuses AI training crawlers
- Site is not served over HTTPS
- robots.txt blocks ChatGPT-User
- robots.txt blocks Claude-SearchBot
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method