Checks / AI crawler access
llms.txt points at pages that crawlers cannot use
The addresses in llms.txt are the pages you tell assistants to read. Some of them are noindex, blocked by robots.txt, redirect, answer an error, or no page links to them, so the map contradicts the site.
llms_txt_conflicts · AI crawler access · Notice
What to change
Remove those addresses from llms.txt, or fix the page: drop the noindex or the robots rule, link to the final address, and link to it from a page of the site.
How common it is
We have not measured how common this is on a sample of sites. Counts appear here when a study covers it.
About this check
| Check ID | llms_txt_conflicts |
|---|---|
| Report section | AI crawler access |
| Severity | Notice |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | No, the fix is described in words |
Related checks
- No llms-full.txt (optional)
- No llms.txt (optional)
- Site is not served over HTTPS
- Pages return an error status
- robots.txt blocks ChatGPT-User
- robots.txt blocks Claude-SearchBot
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method