AI crawler access for nytimes.com

Crawlers that decide whether assistants can cite nytimes.com are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot, Googlebot. Training-only crawlers blocked: GPTBot, ClaudeBot, Google-Extended. That is a legitimate choice and does not remove nytimes.com from answers.

Crawler	Used for	robots.txt	Live request
GPTBot	Model training	Blocked by Disallow: /	Answered 403: refused or challenged
OAI-SearchBot	Search and citations	Blocked by Disallow: /	Answered 403: refused or challenged
ChatGPT-User	Page fetched for a person	Blocked by Disallow: /	Answered 403: refused or challenged
PerplexityBot	Search and citations	Blocked by Disallow: /	Answered 403: refused or challenged
ClaudeBot	Model training	Blocked by Disallow: /	Answered 403: refused or challenged
Google-Extended	Model training	Blocked by Disallow: /	Not requested (robots.txt switch only)
Bingbot	Search and citations	Allowed · 86 path rules restrict this crawler	Answered 403: refused or challenged
Googlebot	Search and citations	Allowed · 93 path rules restrict this crawler	Answered 403: refused or challenged

robots.txt found and read.

Our own CitedBot got 403 on the home page too, so the block is not specific to AI crawlers.

Googlebot, Bingbot and OpenAI's crawlers are also verified by IP address. A refusal for their string from our address can be a site protecting itself from fakes; if the real crawler is blocked, Google Search Console and Bing Webmaster Tools show it.

The exact fix

Edit robots.txt: under User-agent: GPTBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: OAI-SearchBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: ChatGPT-User, remove this line:

Disallow: /

Edit robots.txt: under User-agent: PerplexityBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: ClaudeBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: Google-Extended, remove this line:

Disallow: /

Your server answered GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Bingbot, Googlebot with a refusal when the user-agent was that crawler's. Find the rule that does it (CDN or firewall bot protection, security plugin, server user-agent rule) and let these strings through.

Next: the rest of the site

This tool reads one page and two files. The free check crawls a sample of pages and lists everything that keeps search engines and assistants from reading nytimes.com, with the exact fix for each.

Run the free check on nytimes.com

Method: robots.txt read with the RFC 9309 rules; one GET of https://nytimes.com/ per crawler string, from CitedBy's server (crawlers normally come from their own address ranges). Our own crawler is CitedBot/1.0. Checked 2026-10-21 (UTC).

Who is who

ChatGPT, Perplexity, Gemini, Copilot and Claude answer buyers' questions with different crawlers. Search crawlers (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot) decide whether you can be cited. ChatGPT-User fetches a page when a person asks about it. Training crawlers (GPTBot, ClaudeBot, Google-Extended) feed models, and blocking them is a choice that does not remove you from answers.

A note for EU sites

In the EU, a machine-readable reservation such as a robots.txt rule is how a rightsholder opts out of commercial text-and-data mining (Article 4 of Directive (EU) 2019/790). Blocking a training crawler is therefore a legitimate, documented choice. This is information, not legal advice.

See a real result

Also free: llms.txt checker