Checks / AI crawler access
robots.txt blocks OAI-SearchBot
OAI-SearchBot decides whether ChatGPT search can show your site.
robots_oai_searchbot · AI crawler access · Error
What triggers it
robots.txt does not let OAI-SearchBot fetch your home page (a Disallow rule in its own group, or in the * group when no group names it), and it leaves no other path open for it.
What to change
Remove the Disallow rule for OAI-SearchBot in robots.txt.
Example
Lines that allow OAI-SearchBot in robots.txt:
User-agent: OAI-SearchBot
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: no group that names OAI-SearchBot (or the * group, if none does) disallows the home page
- Run a new crawl: the AI crawler table lists OAI-SearchBot as allowed
About OAI-SearchBot
OAI-SearchBot is OpenAI's crawler for ChatGPT search. Sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. It obeys robots.txt under the token OAI-SearchBot, and OpenAI publishes 39 IP ranges for it.
OpenAI's own wording is that opted-out sites "will not be shown in ChatGPT search answers". That is the cost: a shop that blocks OAI-SearchBot gives up the part of ChatGPT that links to sources. OpenAI recommends the opposite for sites that want to appear: allow OAI-SearchBot in robots.txt and allow requests from its published IP ranges.
Job: Search. Follows robots.txt: Yes, it follows robots.txt.
OAI-SearchBot: operator, user-agent string and address list
How common it is
On a sample, the study measured robots.txt blocking OAI-SearchBot on the homepage's site: 0 of 294 Show HN sites (0%), and 0 of 67 Lovable sites (0%).
Fetched 20 and 21 Oct 2026 as CitedBot, robots.txt obeyed, no JavaScript. Study: Launch crawl 2026-10 (English).
About this check
| Check ID | robots_oai_searchbot |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt blocks Claude-SearchBot
- robots.txt blocks Claude-User
- robots.txt keeps ChatGPT-User out of your home page and most of the site
- robots.txt keeps Claude-SearchBot out of your home page and most of the site
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method