AI crawlers / AI crawlers
PerplexityBot: how it differs from Perplexity-User and how to allow it
What PerplexityBot is
Perplexity names two agents. PerplexityBot builds the search index; Perplexity-User visits a page when a person asks. Perplexity's page contains configuration guidelines for popular WAF providers because a firewall can stop the crawler before robots.txt matters.
| Fact | PerplexityBot |
|---|---|
| Operator | Perplexity |
| Job | Search |
| robots.txt token | PerplexityBot |
| Follows robots.txt | Yes, it follows robots.txt |
| Published address list | 8 ranges (perplexity.com/perplexitybot.json, file dated 2025-02-07) |
| Vendor page read | 21 October 2026 |
What the vendor says
In the vendor's words: “PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.”
Source: Perplexity: Perplexity Crawlers, read on 21 October 2026.
What happens if you block it
Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges if you want to appear in its search results. It also says changes to your robots.txt can take up to 24 hours to be reflected in its systems, so a check made an hour after an edit proves little.
The robots.txt lines
To allow it:
User-agent: PerplexityBot
Allow: /To block it:
User-agent: PerplexityBot
Disallow: /The user-agent string the vendor shows:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)How to tell a real request from a fake one
Perplexity's file at perplexity.com/perplexitybot.json held 8 prefixes on 21 October 2026, creation time 2025-02-07. Its WAF section suggests a rule that combines the user-agent with the IP condition, so a request claiming the name from another address does not get through.
Compare with Perplexity-User
Perplexity-User: The user-triggered fetcher; Perplexity says it generally ignores robots.txt.
Check your own site
The audit box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub.
Questions people ask
How long does a robots.txt change take for PerplexityBot?
Perplexity says it may take up to 24 hours for its systems to reflect a change.
Does PerplexityBot train AI models?
Perplexity says it is not used to crawl content for AI foundation models.
Sources
- Perplexity: Perplexity Crawlers
- perplexity.com/perplexitybot.json, read 21 October 2026
See what AI answers about your shop
The audit crawls your site up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. From €9, 14-day money-back guarantee.
Audit my siteAudit from €9, paid once. 14-day money-back guarantee.