AI crawlers: who they are, what they do, how to allow or block them
One page per AI crawler, with the operator, the robots.txt token, the vendor's own sentence, what blocking costs and the vendor's published address list, each read from the vendor's page on the date shown.
- Amazonbot, Amzn-SearchBot and Amzn-User: three Amazon agents, three rulesAmazon's page now describes three agents with independent robots.txt settings. What each does, which may train Amazon AI models and which may ignore robots.txt.
- Applebot-Extended: opting out of Apple's model training without leaving searchApplebot-Extended is Apple's opt-out token for generative model training. What it does, why Applebot can still crawl you and what stays discoverable in Spotlight, Siri and Safari.
- ChatGPT-User: the fetcher that opens your page when someone asksChatGPT-User visits a page when a person asks ChatGPT or a custom GPT about it. Why robots.txt may not apply, its user-agent string and 234 published IP ranges.
- Claude-SearchBot: Anthropic's search crawler and what blocking costsClaude-SearchBot indexes content to improve Claude's search results. The robots.txt token, what Anthropic says a block does and how it differs from the other two Claude robots.
- Claude-User: the agent that fetches pages when someone asks ClaudeClaude-User retrieves a page when a person asks Claude something. What Anthropic says disabling it does, and how it differs from ClaudeBot and Claude-SearchBot.
- ClaudeBot: Anthropic's training crawler, Crawl-delay and blockingClaudeBot collects web content that could contribute to Anthropic's model training. The robots.txt group, the Crawl-delay Anthropic supports and why IP blocking does not work.
- DuckAssistBot: DuckDuckGo's real-time answer crawler and its 486 addressesDuckAssistBot fetches pages for DuckDuckGo's AI-assisted answers, which cite their sources. Opt-out takes 72 hours, search rankings are unaffected and DuckDuckGo lists 486 IP addresses.
- Google-Extended: the robots.txt token for Gemini, not a crawlerGoogle-Extended is a control token, not a crawler: it has no user agent and never appears in server logs. What it controls, what it leaves alone and why you cannot verify it with an IP list.
- Googlebot: what it covers, its IP ranges and what a block removesGooglebot's robots.txt token controls Google Search and the products built on it. Its user-agent, the 317 IP ranges Google publishes and how to confirm a request is real.
- GPTBot: what it does and what blocking it changesGPTBot crawls content that may be used to train OpenAI models. What a Disallow does, what it leaves alone, its user-agent and the 18 IP ranges OpenAI publishes.
- MistralAI-User, MistralAI-Index and MistralAI-Training explainedMistral documents three robots: a user-action fetcher for Vibe, a search index agent and a training crawler. What each does, its robots.txt token and the IP files it publishes.
- OAI-SearchBot: what it is and how to allow or block itOAI-SearchBot decides whether your pages can appear in ChatGPT search answers. Its robots.txt token, user-agent, 39 published IP ranges and what a block costs you.
- PerplexityBot: how it differs from Perplexity-User and how to allow itPerplexityBot surfaces and links sites in Perplexity search. It is not used for foundation model crawling. User-agent, the 8 IP ranges Perplexity publishes and WAF advice.