AI crawlers / AI crawlers
ExaSearchBot: what it does and what blocking it changes
What ExaSearchBot is
ExaSearchBot is the crawler of Exa, a search engine. Exa says its purpose is search and retrieval, so pages become discoverable and can be cited. Pages Exa fetches on a user's request rather than by crawling go out as ExaUser.
| Fact | ExaSearchBot |
|---|---|
| Operator | Exa |
| Job | Search and retrieval index |
| robots.txt token | ExaSearchBot |
| Follows robots.txt | Yes, vendor says it checks robots.txt before fetching |
| IP addresses | Verify tool |
| Vendor page read | 24 October 2026 |
What the vendor says
In the vendor's words (English original): “Exa is a search engine. Similar to other search engines, Exa uses a robot (ExaSearchBot) to discover and index pages on the public web.”
Source: Exa: Exa Search Crawler, read on 24 October 2026.
What happens if you block ExaSearchBot
Exa says User-agent: ExaSearchBot with Disallow: / blocks ExaSearchBot from the whole site. If there is no ExaSearchBot group, it follows the rules set for major search engines, otherwise the * group. The vendor says robots.txt controls crawling, not indexing, so already indexed pages are removed with a noindex meta tag or X-Robots-Tag header.
The robots.txt lines
To allow ExaSearchBot:
User-agent: ExaSearchBot
Allow: /To block ExaSearchBot:
User-agent: ExaSearchBot
Disallow: /Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers.
The user-agent string the vendor shows:
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)How to tell a real request from a fake one
The vendor says every request is cryptographically signed with HTTP Message Signatures (Web Bot Auth) and gives a public key directory to verify against, and that verification holds regardless of IP address.
Compare with PerplexityBot
PerplexityBot: Both are search-index crawlers that feed AI answers with cited sources; ExaSearchBot signs its requests with Web Bot Auth and falls back to major search engines' rules when it has no group of its own.
Your own site
The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). ExaSearchBot is not one of them: read your robots.txt for a ExaSearchBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub.
Questions people ask
Does ExaSearchBot follow robots.txt?
Yes, vendor says it checks robots.txt before fetching. Give ExaSearchBot its own User-agent group in robots.txt: the operator's other crawlers have their own settings.
How do I know ExaSearchBot is really blocked?
robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network.
Sources
See what AI answers about your shop
The audit crawls your site up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. From €9, 14-day money-back guarantee.
Audit my siteAudit from €9, paid once. 14-day money-back guarantee.