Checks / AI crawler access
robots.txt blocks Claude-User
Claude-User fetches a page when a person asks that assistant about it.
robots_claude_user · AI crawler access · Error
What triggers it
robots.txt does not let Claude-User fetch your home page (a Disallow rule in its own group, or in the * group when no group names it), and it leaves no other path open for it.
What to change
Remove the Disallow rule for Claude-User in robots.txt.
Example
Lines that allow Claude-User in robots.txt:
User-agent: Claude-User
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: no group that names Claude-User (or the * group, if none does) disallows the home page
- Run a new crawl: the AI crawler table lists Claude-User as allowed
About Claude-User
Claude-User is the agent Claude may use to access a website when a person asks it a question. Anthropic says disabling it prevents retrieval of your content in response to a user query, which may reduce your visibility for user-directed web search. It is separate from ClaudeBot and Claude-SearchBot.
This is where Anthropic's wording on cost is most direct: disabling Claude-User prevents retrieval "in response to a user query" and "may reduce your site's visibility for user-directed web search". A shop that blocks it can still be known to the model from other sources, but Claude cannot open the page when asked.
Job: User-triggered fetch. Follows robots.txt: Yes, Anthropic says its robots respect robots.txt.
Claude-User: operator, user-agent string and address list
How common it is
We have not measured how common this is on a sample of sites. Counts appear here when a study covers it.
About this check
| Check ID | robots_claude_user |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt blocks ChatGPT-User
- robots.txt blocks Claude-SearchBot
- robots.txt blocks OAI-SearchBot
- robots.txt keeps ChatGPT-User out of your home page and most of the site
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method