Checks / AI crawler access
robots.txt keeps ClaudeBot out of your home page and most of the site
ClaudeBot collects training data and may read only the paths robots.txt allows for it. This is a choice: most of your pages stay out of future model training, not out of AI search.
robots_partial_claudebot · AI crawler access · Notice
What triggers it
robots.txt does not let ClaudeBot fetch your home page, but leaves some other paths open for it; the finding names the first three.
What to change
Remove the Disallow: / rule for ClaudeBot in robots.txt, or add Allow rules for the other paths you want read.
Example
Lines that allow ClaudeBot in robots.txt:
User-agent: ClaudeBot
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: the group that names ClaudeBot (or the * group, if none does) leaves most of the site open, the home page included
- Run a new crawl: the AI crawler table no longer reports ClaudeBot as mostly blocked
About ClaudeBot
ClaudeBot is Anthropic's crawler for content that could contribute to model training. A robots.txt Disallow signals that a site's future materials should be excluded from training datasets. Anthropic honours Crawl-delay, and warns that blocking its IP addresses can stop it reading robots.txt, so use robots.txt instead.
A separate search robot: blocking ClaudeBot does not touch it.
Anthropic's article says its robots respect "do not crawl" signals in robots.txt and will not try to bypass CAPTCHAs. It supports the non-standard Crawl-delay directive, for example `User-agent: ClaudeBot` followed by `Crawl-delay: 1`. It also says that blocking by IP address may not give a lasting opt-out, because it stops the robot from reading robots.txt. Anthropic's article states that robots.txt must be set for every subdomain you want to opt out.
Job: Training. Follows robots.txt: Yes, it follows robots.txt and Crawl-delay.
ClaudeBot: operator, user-agent string and address list
How common it is
We have not measured how common this is on a sample of sites. Counts appear here when a study covers it.
About this check
| Check ID | robots_partial_claudebot |
|---|---|
| Report section | AI crawler access |
| Severity | Notice |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt blocks GPTBot
- robots.txt keeps CCBot out of your home page and most of the site
- robots.txt keeps Google-Extended out of your home page and most of the site
- robots.txt keeps GPTBot out of your home page and most of the site
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method