AI crawlers / AI crawlers
GPTBot: what it does and what blocking it changes
What GPTBot is
GPTBot is the crawler people usually mean when they say "block OpenAI". OpenAI's page says it is used to crawl content that may be used in training its models. It is one of three crawlers named in the OpenAI documentation for the ChatGPT product family, next to OAI-SearchBot and ChatGPT-User.
| Fact | GPTBot |
|---|---|
| Operator | OpenAI |
| Job | Training |
| robots.txt token | GPTBot |
| Follows robots.txt | Yes, it follows robots.txt |
| Published address list | 18 ranges (openai.com/gptbot.json, file dated 2026-09-22) |
| Vendor page read | 21 October 2026 |
What the vendor says
In the vendor's words: “GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.”
Source: OpenAI: Overview of OpenAI Crawlers, read on 21 October 2026.
What happens if you block it
OpenAI frames a GPTBot Disallow as a signal about training: the content should not be used in training generative AI foundation models. The same page gives an example of a site allowing OAI-SearchBot to appear in search results while disallowing GPTBot, so a block here does not by itself take you out of ChatGPT's search answers. Whether to block is a business choice; both are legitimate.
The robots.txt lines
To allow it:
User-agent: GPTBot
Allow: /To block it:
User-agent: GPTBot
Disallow: /The user-agent string the vendor shows:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotHow to tell a real request from a fake one
The GPTBot list at openai.com/gptbot.json is the shortest of the three OpenAI files we counted: 18 prefixes on 21 October 2026, creation time 2026-09-22. When a request names GPTBot but comes from an address outside that list, the name alone proves nothing.
Compare with OAI-SearchBot
OAI-SearchBot: Allow it and disallow GPTBot to stay in ChatGPT search while opting out of training.
Check your own site
The audit box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub.
Questions people ask
If I block GPTBot, will ChatGPT stop mentioning my shop?
Not through GPTBot alone. OpenAI treats its crawler settings as independent and says search appearance is managed through OAI-SearchBot. Check both tokens in your robots.txt.
How do I know GPTBot is really blocked?
Robots.txt is only one layer. A CDN or firewall rule can also turn the crawler away. Our guide to the four places GPTBot can be stopped shows a two-request test for each.
Sources
- OpenAI: Overview of OpenAI Crawlers
- openai.com/gptbot.json, read 21 October 2026
See what AI answers about your shop
The audit crawls your site up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. From €9, 14-day money-back guarantee.
Audit my siteAudit from €9, paid once. 14-day money-back guarantee.