Checks / Links, canonical and speed
Pages refuse or do not answer our crawler
Some pages of the site answered 429 (too many requests), refused the crawler (401, 403 or a challenge page), or did not answer in time on a second try. Search engines and AI crawlers meet the same walls: repeated 429 or timeouts make them slow down or stop crawling the site, and refused pages are not read. We left these pages out of the page checks, so this is the only place they show.
crawl_refused · Links, canonical and speed · Warning
What to change
Match the fix to what the page did. 429: serve the pages from a cache or a static result and raise the rate limit for crawlers; keep Retry-After short. 401, 403 or a challenge page: allow CitedBot in the firewall rule that blocks it; the bot test on this report shows which crawlers get through. No answer: find the route that computes a live result on the first request and serve a stored result or a form instead. If verified Googlebot and OAI-SearchBot get a 200 on these pages and only our crawler is refused, the wall is a rule against unknown bots, and you can leave it.
How common it is
In the audits of our own sites on 26 Oct 2026 (4 sites): finding judged right on citedby.eu.
Source: finding audit 2026-10-26 (English).
About this check
| Check ID | crawl_refused |
|---|---|
| Report section | Links, canonical and speed |
| Severity | Warning |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | No, the fix is described in words |
Related checks
- Broken internal links
- Canonical points to a broken or redirecting URL
- Redirect loop
- Canonical points to a noindex page
- Canonical points to another page
- Very large HTML
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method