AI crawlers / AI crawlers

Claude-SearchBot: Anthropic's search crawler and what blocking costs

By the CitedBy editorial team · Reviewed on 2026-10-21

What Claude-SearchBot is

Claude-SearchBot is the indexing side of Anthropic's search. Anthropic describes it as analysing online content to improve relevance and accuracy of search responses. ClaudeBot collects for training and Claude-User answers a live question.

What the vendor says

In the vendor's words: “Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”

Source: Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026), read on 21 October 2026.

What happens if you block it

Anthropic names two effects of disabling it: your content is not indexed for search optimization, and your site's visibility and accuracy in user search results may fall.

The robots.txt lines

To allow it:

User-agent: Claude-SearchBot
Allow: /

To block it:

User-agent: Claude-SearchBot
Disallow: /

How to tell a real request from a fake one

The help article is dated 7 April 2026. It lists no user-agent string, so a log line that names Claude-SearchBot is a claim you cannot confirm from the article alone. Anthropic says traffic from an address on its linked list is Anthropic's, and it asks site owners who think a robot is misbehaving to write from an email on the domain concerned.

Compare with ClaudeBot

ClaudeBot: Blocking ClaudeBot opts out of training only; it does not remove you from Claude's search index.

Check your own site

The audit box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub.

Questions people ask

Do I have to allow Claude-SearchBot to be cited by Claude?

Anthropic says that disabling it may reduce your visibility and accuracy in user search results. It makes no promise either way that allowing it leads to a citation.

Where do I put the rule?

In the robots.txt at the top level of each host. Anthropic says to do this for every subdomain.

Sources

• Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026)