AI crawler access for cdiscount.com
All 8 crawlers can read cdiscount.com: robots.txt allows them and the server answered each test request.
| Crawler | Used for | robots.txt | Live request |
|---|---|---|---|
| GPTBot | Model training | Allowed · 244 path rules restrict this crawler | Answered 200 |
| OAI-SearchBot | Search and citations | Allowed · 244 path rules restrict this crawler | Answered 200 |
| ChatGPT-User | Page fetched for a person | Allowed · 244 path rules restrict this crawler | Answered 200 |
| PerplexityBot | Search and citations | Allowed · 244 path rules restrict this crawler | Answered 200 |
| ClaudeBot | Model training | Allowed · 244 path rules restrict this crawler | Answered 200 |
| Google-Extended | Model training | Allowed · 244 path rules restrict this crawler | Not requested (robots.txt switch only) |
| Bingbot | Search and citations | Allowed · 244 path rules restrict this crawler | Answered 200 |
| Googlebot | Search and citations | Allowed · 244 path rules restrict this crawler | Answered 200 |
The exact fix
Nothing to fix for crawler access.
Next: the rest of the site
This tool reads one page and two files. The free check crawls a sample of pages and lists everything that keeps search engines and assistants from reading cdiscount.com, with the exact fix for each.
Run the free check on cdiscount.com
Who is who
ChatGPT, Perplexity, Gemini, Copilot and Claude answer buyers' questions with different crawlers. Search crawlers (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot) decide whether you can be cited. ChatGPT-User fetches a page when a person asks about it. Training crawlers (GPTBot, ClaudeBot, Google-Extended) feed models, and blocking them is a choice that does not remove you from answers.
A note for EU sites
In the EU, a machine-readable reservation such as a robots.txt rule is how a rightsholder opts out of commercial text-and-data mining (Article 4 of Directive (EU) 2019/790). Blocking a training crawler is therefore a legitimate, documented choice. This is information, not legal advice.