Checks / AI crawler access
robots.txt blocks ChatGPT-User
ChatGPT-User fetches a page when a person asks ChatGPT about it. Blocked, ChatGPT cannot open your pages on request.
robots_chatgpt_user · AI crawler access · Error
What triggers it
robots.txt does not let ChatGPT-User fetch your home page (a Disallow rule in its own group, or in the * group when no group names it), and it leaves no other path open for it.
What to change
Remove the Disallow rule for ChatGPT-User in robots.txt.
Example
Lines that allow ChatGPT-User in robots.txt:
User-agent: ChatGPT-User
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: no group that names ChatGPT-User (or the * group, if none does) disallows the home page
- Run a new crawl: the AI crawler table lists ChatGPT-User as allowed
About ChatGPT-User
ChatGPT-User visits a web page when a person asks ChatGPT or a custom GPT a question that needs it. OpenAI says it is not used to crawl automatically and that, because a user starts the request, robots.txt rules may not apply. OpenAI lists 234 IP ranges for it.
Because the visit is user-initiated, OpenAI warns that robots.txt rules may not apply, so a `Disallow` line for ChatGPT-User is a request that user-initiated visits may not follow. OpenAI also says ChatGPT-User is not used to decide whether content appears in search and sends search opt-outs to OAI-SearchBot. If you need to stop these requests, the control is on your server or CDN, not in robots.txt alone.
Job: User-triggered fetch. Follows robots.txt: Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User, because its actions are initiated by a user..
ChatGPT-User: operator, user-agent string and address list
How common it is
On a sample, the study measured robots.txt blocking ChatGPT-User on the homepage's site: 0 of 294 Show HN sites (0%), and 0 of 67 Lovable sites (0%).
Fetched 20 and 21 Oct 2026 as CitedBot, robots.txt obeyed, no JavaScript. Study: Launch crawl 2026-10 (English).
About this check
| Check ID | robots_chatgpt_user |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- Site is not served over HTTPS
- Pages return an error status
- robots.txt blocks Claude-SearchBot
- robots.txt blocks Claude-User
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method