CitedBy / Data / AI access / WordPress
AI-Access reference · read 26 Oct 2026 · first reading 23 Oct 2026
WordPress sites in the EU: which AI crawlers they block
On 26 Oct 2026, 10.4% of 1,655 popular EU websites built on WordPress asked at least one of GPTBot, ClaudeBot, Google-Extended and CCBot to stay out in robots.txt (9.0 to 12.0), rank 4 of the 6 CMSs with at least 100 sites read counting from the most blocking; 9.2% blocked GPTBot, 4.4% blocked at least one answer-engine search crawler, and across all 25 countries together 18.5% did.
What this measures
- 10.4% (172 of 1,655) block at least one of GPTBot, ClaudeBot, Google-Extended and CCBot on 26 Oct 2026; 10.1% did on 23 Oct 2026. All four: 5.4% (89).
- 9.2% block GPTBot (152); 7.9% block ClaudeBot; 5.9% block Google-Extended; 8.3% block CCBot.
- 4.4% (73) block at least one answer-engine search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Googlebot, the control: 0.4%.
- llms.txt: 15.2% (248 of 1,632 that answered, 95% interval 13.5 to 17.0) publish a valid llms.txt, against 11.7% of all 10,223 sites that answered (read 25 Oct 2026).
Between 23 Oct 2026 and 26 Oct 2026, 8 of the 1,655 sites read in both runs changed what they ask at least one of the 14 AI crawlers to do (8 started blocking, 0 stopped). Next to it in the ranking: Magento (11.2%), Drupal (9.3%). See all countries and CMSs.
Running WordPress? The CitedBy WordPress plugin runs these checks on your own site, and WordPress.com has its own steps.
By crawler
| Crawler | Block it, 23 Oct 2026 (n 1,667) | Block it, 26 Oct 2026 (n 1,655) | 95% interval | All 25 countries, 26 Oct 2026 |
|---|---|---|---|---|
| GPTBot | 8.9% (148) | 9.2% (152) | 7.9 to 10.7 | 15.5% |
| OAI-SearchBot | 2.6% (44) | 2.6% (43) | 1.9 to 3.5 | 5.4% |
| ChatGPT-User | 4.3% (71) | 4.2% (70) | 3.4 to 5.3 | 7.3% |
| ClaudeBot | 7.6% (126) | 7.9% (130) | 6.7 to 9.3 | 13.4% |
| Claude-SearchBot | 1.9% (31) | 1.8% (30) | 1.3 to 2.6 | 4.0% |
| Claude-User | 1.7% (29) | 1.7% (28) | 1.2 to 2.4 | 3.9% |
| PerplexityBot | 4.0% (67) | 4.0% (67) | 3.2 to 5.1 | 7.1% |
| Perplexity-User | 1.9% (32) | 1.9% (31) | 1.3 to 2.6 | 4.2% |
| Google-Extended | 5.5% (91) | 5.9% (97) | 4.8 to 7.1 | 11.4% |
| Applebot-Extended | 5.9% (99) | 6.2% (103) | 5.2 to 7.5 | 10.8% |
| CCBot | 8.0% (134) | 8.3% (138) | 7.1 to 9.8 | 15.3% |
| Bytespider | 7.7% (128) | 7.9% (130) | 6.7 to 9.3 | 14.4% |
| Amazonbot | 7.5% (125) | 7.7% (128) | 6.5 to 9.1 | 13.6% |
| meta-externalagent | 5.9% (98) | 6.2% (102) | 5.1 to 7.4 | 12.1% |
| Googlebot (control) | 0.5% (8) | 0.4% (7) | 0.2 to 0.9 | 0.8% |
| Bingbot (control) | 1.0% (16) | 0.9% (15) | 0.6 to 1.5 | 1.5% |
"Blocks" means the crawler may not fetch / under the site's robots.txt. Googlebot and Bingbot are controls: sites that shut out AI crawlers almost never shut out ordinary search.
Method and limits
- Sample: the top 600 domains by Tranco rank under each of 25 EU country-code domains (Luxembourg 250), Tranco list 56Z8N. The same sites are read again each week, so the dates on this page are repeat readings of one list, not new samples.
- What is read:
/robots.txtof each site, per RFC 9309; "blocks" means the crawler may not fetch/. Shares use the sites whose robots.txt was read as base; the others are coverage, not "allows". llms.txt: the file at/llms.txtis valid if it passes the llmstxt.org shape check (method). - Limits: robots.txt states a request, not what a firewall enforces, and this index does not measure what any assistant cites. A country-code domain is not the operator's country, and a CMS is a signature in the home page, not a verified platform. Popular sites only: a random sample of 24,550 .de, .at and .ch hosts reads lower, 2.27% for GPTBot (Ahrefs, Aug 2026).
- Data: One row per domain, 23 Oct 2026 · One row per domain, 26 Oct 2026 · llms.txt per domain, 25 Oct 2026. Licence CC BY 4.0; cite as "CitedBy EU AI-Access Index, 26 Oct 2026".
What does your robots.txt say to AI crawlers?
The free check shows which AI crawlers your robots.txt and your server let in, with the exact fix. With the paid audit, we crawl your site up to your plan's page limit and give you the exact fix for each issue. From €9, 14-day money-back guarantee.