AI crawler access for nu.nl

Crawlers that decide whether assistants can cite nu.nl are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot, Googlebot. Training-only crawlers blocked: GPTBot, ClaudeBot, Google-Extended. That is a legitimate choice and does not remove nu.nl from answers.

Crawler	Used for	robots.txt	Live request
GPTBot	Model training	Blocked by Disallow: /	Answered 403: refused or challenged
OAI-SearchBot	Search and citations	Blocked by Disallow: /	Answered 403: refused or challenged
ChatGPT-User	Page fetched for a person	Blocked by Disallow: /	Answered 403: refused or challenged
PerplexityBot	Search and citations	Blocked by Disallow: /	Answered 403: refused or challenged
ClaudeBot	Model training	Blocked by Disallow: /	Answered 403: refused or challenged
Google-Extended	Model training	Blocked by Disallow: /	Not requested (robots.txt switch only)
Bingbot	Search and citations	Allowed · 6 path rules restrict this crawler	Answered 403: refused or challenged
Googlebot	Search and citations	Allowed · 6 path rules restrict this crawler	Answered 403: refused or challenged

robots.txt found and read.

Googlebot, Bingbot and OpenAI's crawlers are also verified by IP address. A refusal for their string from our address can be a site protecting itself from fakes; if the real crawler is blocked, Google Search Console and Bing Webmaster Tools show it.

The exact fix

Edit robots.txt: under User-agent: GPTBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: OAI-SearchBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: ChatGPT-User, remove this line:

Disallow: /

Edit robots.txt: under User-agent: PerplexityBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: ClaudeBot, remove this line:

Disallow: /

Edit robots.txt: under User-agent: Google-Extended, remove this line:

Disallow: /

Your server answered GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Bingbot, Googlebot with a refusal when the user-agent was that crawler's. Find the rule that does it (CDN or firewall bot protection, security plugin, server user-agent rule) and let these strings through.

Next: the rest of the site

This tool reads one page and two files. The free check crawls a sample of pages and lists everything that keeps search engines and assistants from reading nu.nl, with the exact fix for each.

Run the free check on nu.nl

Method: robots.txt read with the RFC 9309 rules; one GET of https://nu.nl/ per crawler string, from CitedBy's server (crawlers normally come from their own address ranges). Our own crawler is CitedBot/1.0. Checked 2026-10-21 (UTC).

Who is who

ChatGPT, Perplexity, Gemini, Copilot and Claude answer buyers' questions with different crawlers. Search crawlers (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot) decide whether you can be cited. ChatGPT-User fetches a page when a person asks about it. Training crawlers (GPTBot, ClaudeBot, Google-Extended) feed models, and blocking them is a choice that does not remove you from answers.

A note for EU sites

In the EU, a machine-readable reservation such as a robots.txt rule is how a rightsholder opts out of commercial text-and-data mining (Article 4 of Directive (EU) 2019/790). Blocking a training crawler is therefore a legitimate, documented choice. This is information, not legal advice.

See a real result

Also free: llms.txt checker