Checks / AI crawler access
robots.txt blocks GPTBot
Blocks GPTBot, which collects training data. This keeps your pages out of future model training, not out of ChatGPT or other AI search.
robots_gptbot · AI crawler access · Notice
What triggers it
robots.txt does not let GPTBot fetch your home page (a Disallow rule in its own group, or in the * group when no group names it), and it leaves no other path open for it.
What to change
Only if you want your pages in future model training: remove the Disallow rule for GPTBot. ChatGPT search is governed by OAI-SearchBot.
Example
Lines that allow GPTBot in robots.txt:
User-agent: GPTBot
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: no group that names GPTBot (or the * group, if none does) disallows the home page
- Run a new crawl: the AI crawler table lists GPTBot as allowed
About GPTBot
GPTBot is OpenAI's training crawler. Disallowing it in robots.txt tells OpenAI the site's content should not be used to train its generative AI foundation models. It does not control ChatGPT search, which uses OAI-SearchBot. OpenAI publishes 18 IP ranges for GPTBot.
OpenAI frames a GPTBot Disallow as a signal about training: the content should not be used in training generative AI foundation models. The same page gives an example of a site allowing OAI-SearchBot to appear in search results while disallowing GPTBot, so a block here does not by itself take you out of ChatGPT's search answers. Whether to block is a business choice; both are legitimate.
Job: Training. Follows robots.txt: Yes, it follows robots.txt.
GPTBot: operator, user-agent string and address list
How common it is
On a sample, the study measured robots.txt blocking GPTBot on the homepage's site: 12 of 294 Show HN sites (4%), and 0 of 67 Lovable sites (0%).
Fetched 20 and 21 Oct 2026 as CitedBot, robots.txt obeyed, no JavaScript. Study: Launch crawl 2026-10 (English).
About this check
| Check ID | robots_gptbot |
|---|---|
| Report section | AI crawler access |
| Severity | Notice |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt blocks ClaudeBot
- robots.txt blocks Google-Extended
- robots.txt keeps CCBot out of your home page and most of the site
- robots.txt keeps ClaudeBot out of your home page and most of the site
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method