Checks / AI crawler access
Your firewall or CDN refuses AI search crawlers
Our test request with an AI search crawler's user agent was refused (403, 429, challenge page or timeout) while a browser got the page. Real crawlers may still get through if your CDN verifies them by IP, so check your CDN logs.
waf_blocks_bots · AI crawler access · Error
What triggers it
A request for your home page that names CitedBot, PerplexityBot, OAI-SearchBot or Claude-SearchBot in its user agent gets something other than 200 (or no answer within 15 seconds, twice) while a browser request gets 200.
What to change
In Cloudflare, Akamai, Sucuri or your host's WAF, allow the verified AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) or turn off 'Block AI bots'.
How to confirm the fix
- In your CDN or firewall logs, find requests from the crawler's user agent and confirm they get 200
- Run a new crawl: the live test column shows 200
How common it is
On a sample, the study measured a firewall refusing a live request from a search crawler on the homepage: 3 of 293 Show HN sites (1%), and 0 of 66 Lovable sites (0%).
Fetched 20 and 21 Oct 2026 as CitedBot, robots.txt obeyed, no JavaScript. Study: Launch crawl 2026-10 (English).
About this check
| Check ID | waf_blocks_bots |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | No, the fix is described in words |
Related checks
- robots.txt blocks Perplexity-User
- robots.txt blocks PerplexityBot
- No llms-full.txt (optional)
- llms.txt points at pages that crawlers cannot use
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method