Checks / AI crawler access
robots.txt blocks Perplexity-User
Perplexity-User fetches a page when a person asks Perplexity about it.
robots_perplexity_user · AI crawler access · Error
What triggers it
robots.txt does not let Perplexity-User fetch your home page (a Disallow rule in its own group, or in the * group when no group names it), and it leaves no other path open for it.
What to change
Remove the Disallow rule for Perplexity-User in robots.txt.
Example
Lines that allow Perplexity-User in robots.txt:
User-agent: Perplexity-User
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: no group that names Perplexity-User (or the * group, if none does) disallows the home page
- Run a new crawl: the AI crawler table lists Perplexity-User as allowed
About Perplexity-User
Perplexity-User is Perplexity's user-action fetcher: when a user asks a question, it may visit a page to answer accurately and link to it. Perplexity says it is not used for web crawling or for training AI models, and that it generally ignores robots.txt rules because a user requested the fetch.
Perplexity lists Perplexity-User as a robots.txt token and says it controls which sites these user requests can access. Perplexity also says this fetcher generally ignores robots.txt rules because a user requested the fetch. Perplexity says each of its robots.txt settings works independently and changes may take up to 24 hours to take effect.
Job: User-triggered fetch. Follows robots.txt: Generally not. Perplexity says this fetcher generally ignores robots.txt rules because a user requested the fetch..
Perplexity-User: operator, user-agent string and address list
How common it is
On a sample, the study measured robots.txt blocking Perplexity-User on the homepage's site: 0 of 294 Show HN sites (0%), and 0 of 67 Lovable sites (0%).
Fetched 20 and 21 Oct 2026 as CitedBot, robots.txt obeyed, no JavaScript. Study: Launch crawl 2026-10 (English).
About this check
| Check ID | robots_perplexity_user |
|---|---|
| Report section | AI crawler access |
| Severity | Error |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt keeps Perplexity-User out of your home page and most of the site
- robots.txt keeps PerplexityBot out of your home page and most of the site
- robots.txt blocks PerplexityBot
- Your firewall or CDN refuses AI search crawlers
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method