Checks / AI crawler access
robots.txt keeps OAI-SearchBot out of your home page and most of the site
OAI-SearchBot may read only the paths robots.txt allows for it. Your home page and the rest of the site stay closed to it, so it cannot cite them.
robots_partial_oai_searchbot · AI crawler access · Error
What triggers it
robots.txt does not let OAI-SearchBot fetch your home page, but leaves some other paths open for it; the finding names the first three.
What to change
Remove the Disallow: / rule for OAI-SearchBot in robots.txt, or add Allow rules for the other paths you want read.
Example
Lines that allow OAI-SearchBot in robots.txt:
User-agent: OAI-SearchBot
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: the group that names OAI-SearchBot (or the * group, if none does) leaves most of the site open, the home page included
- Run a new crawl: the AI crawler table no longer reports OAI-SearchBot as mostly blocked
About OAI-SearchBot
OAI-SearchBot is OpenAI's crawler for ChatGPT search. Sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. It obeys robots.txt under the token OAI-SearchBot, and OpenAI publishes 39 IP ranges for it.
GPTBot is for training; blocking it does not remove you from ChatGPT search because the settings are independent.
OpenAI publishes the addresses OAI-SearchBot uses at openai.com/searchbot.json. On 21 October 2026 that file held 39 prefixes and carried the creation time 2026-01-02. Match the address in your server log against those prefixes before you trust a request that only claims the name in its user-agent. OpenAI also says that when it fetches robots.txt it may add a `robots.txt` marker to the user-agent string, so those requests are easy to tell apart in logs without paths.
Job: Search. Follows robots.txt: Yes, it follows robots.txt.
OAI-SearchBot: operator, user-agent string and address list
How common it is
We have not measured how common this is on a sample of sites. Counts appear here when a study covers it.
About this check
| Check ID | robots_partial_oai_searchbot |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt keeps Claude-SearchBot out of your home page and most of the site
- robots.txt keeps Claude-User out of your home page and most of the site
- robots.txt keeps Perplexity-User out of your home page and most of the site
- robots.txt keeps PerplexityBot out of your home page and most of the site
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method