AI crawler access for theguardian.com

Crawlers that decide whether assistants can cite theguardian.com are blocked: PerplexityBot. Training-only crawlers blocked: ClaudeBot. That is a legitimate choice and does not remove theguardian.com from answers.

Crawler	Used for	robots.txt	Live request
GPTBot	Model training	Allowed · 48 path rules restrict this crawler	Answered 200
OAI-SearchBot	Search and citations	Allowed · 48 path rules restrict this crawler	Answered 200
ChatGPT-User	Page fetched for a person	Allowed · 48 path rules restrict this crawler	Answered 200
PerplexityBot	Search and citations	Blocked by Disallow: /	Answered 403: refused or challenged
ClaudeBot	Model training	Blocked by Disallow: /	Answered 403: refused or challenged
Google-Extended	Model training	Allowed · 48 path rules restrict this crawler	Not requested (robots.txt switch only)
Bingbot	Search and citations	Allowed · 48 path rules restrict this crawler	Answered 200
Googlebot	Search and citations	Allowed · 48 path rules restrict this crawler	Answered 200

robots.txt found and read.

Googlebot, Bingbot and OpenAI's crawlers are also verified by IP address. A refusal for their string from our address can be a site protecting itself from fakes; if the real crawler is blocked, Google Search Console and Bing Webmaster Tools show it.

The exact fix

Edit robots.txt: under User-agent: PerplexityBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: ClaudeBot, remove this line:

Disallow: /

Your server answered PerplexityBot, ClaudeBot with a refusal when the user-agent was that crawler's. Find the rule that does it (CDN or firewall bot protection, security plugin, server user-agent rule) and let these strings through.

Next: the rest of the site

This tool reads one page and two files. The free check crawls a sample of pages and lists everything that keeps search engines and assistants from reading theguardian.com, with the exact fix for each.

Run the free check on theguardian.com

Method: robots.txt read with the RFC 9309 rules; one GET of https://theguardian.com/ per crawler string, from CitedBy's server (crawlers normally come from their own address ranges). Our own crawler is CitedBot/1.0. Checked 2026-10-21 (UTC).

Who is who

ChatGPT, Perplexity, Gemini, Copilot and Claude answer buyers' questions with different crawlers. Search crawlers (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot) decide whether you can be cited. ChatGPT-User fetches a page when a person asks about it. Training crawlers (GPTBot, ClaudeBot, Google-Extended) feed models, and blocking them is a choice that does not remove you from answers.

A note for EU sites

In the EU, a machine-readable reservation such as a robots.txt rule is how a rightsholder opts out of commercial text-and-data mining (Article 4 of Directive (EU) 2019/790). Blocking a training crawler is therefore a legitimate, documented choice. This is information, not legal advice.

See a real result

Also free: llms.txt checker