AI-Access Index · run 1 · read 23 Oct 2026
EU AI-Access Index: which AI crawlers popular EU sites block
On 23 October 2026, 15.3% of 10,426 popular EU websites (top domains under 25 country-code TLDs, Tranco list 56Z8N) asked OpenAI's GPTBot to stay out in robots.txt; 18.4% blocked at least one of GPTBot, ClaudeBot, Google-Extended and CCBot (CitedBy AI-Access Index). 1 in 13 (7.9%) of the same sites blocks at least one crawler that builds AI answer indexes (OAI-SearchBot, Claude-SearchBot or PerplexityBot); 1.0% block Googlebot. German-domain sites block AI training crawlers most often (31.2% block at least one of four), Slovenian-domain sites least (11.2%).
Run 1 · robots.txt read 23 Oct 2026 · Tranco list 56Z8N · 10,426 of 14,650 sites read · This is the latest run. Download the per-site CSV
What this measures
This index records what each site's robots.txt asks AI crawlers to do. It does not measure what any assistant cites or links, and it does not show whether a crawler obeys the file.
- 15.3% (1,596 sites) block GPTBot; 13.3% block ClaudeBot, 11.3% Google-Extended, 15.1% CCBot.
- 18.4% (1,920) block at least one of those four; 9.7% (1,015) block all four.
- 7.9% (823) block at least one answer-engine search crawler (OAI-SearchBot 5.6%, PerplexityBot 7.3%, Claude-SearchBot 4.2%).
- 1.0% block Googlebot, the control: sites that shut out AI crawlers almost never shut out ordinary search.
Shares use the 10,426 sites whose robots.txt we could read as base. "Blocks" means the crawler may not fetch the home page path / under the file's rules.
By crawler
| Crawler | Share of sites that block it | Share that name it in robots.txt |
|---|---|---|
| GPTBot | 15.3% (1,596) | 18.0% |
| OAI-SearchBot | 5.6% (584) | 8.2% |
| ChatGPT-User | 7.5% (778) | 10.1% |
| ClaudeBot | 13.3% (1,386) | 15.7% |
| Claude-SearchBot | 4.2% (441) | 4.7% |
| Claude-User | 4.1% (427) | 4.2% |
| PerplexityBot | 7.3% (763) | 10.0% |
| Perplexity-User | 4.5% (466) | 4.5% |
| Google-Extended | 11.3% (1,173) | 12.8% |
| Applebot-Extended | 10.6% (1,110) | 10.3% |
| CCBot | 15.1% (1,579) | 15.2% |
| Bytespider | 14.3% (1,486) | 13.4% |
| Amazonbot | 13.6% (1,414) | 13.5% |
| meta-externalagent | 11.9% (1,243) | 12.3% |
| Googlebot | 1.0% (108) | 7.0% |
| Bingbot | 1.7% (176) | 7.1% |
Googlebot and Bingbot are controls. "Names it" means robots.txt has a group addressed to that crawler: the owner chose on purpose. Download by crawler (CSV)
By country-code domain
| Country (ccTLD) | Sites read | Block GPTBot | Block at least one of GPTBot, ClaudeBot, Google-Extended, CCBot (95% interval) | Block an answer-engine search crawler |
|---|---|---|---|---|
| Germany (.de) | 464 | 27.6% | 31.2% (27.2 to 35.6) | 16.6% |
| Italy (.it) | 431 | 23.0% | 29.0% (24.9 to 33.5) | 16.5% |
| Poland (.pl) | 427 | 23.4% | 25.8% (21.8 to 30.1) | 7.0% |
| Finland (.fi) | 396 | 22.5% | 25.0% (21.0 to 29.5) | 13.9% |
| France (.fr) | 439 | 20.7% | 24.4% (20.6 to 28.6) | 13.2% |
| Denmark (.dk) | 434 | 20.5% | 22.8% (19.1 to 27.0) | 10.1% |
| Netherlands (.nl) | 430 | 21.6% | 22.8% (19.1 to 27.0) | 15.6% |
| Spain (.es) | 429 | 18.9% | 22.8% (19.1 to 27.0) | 12.1% |
| Sweden (.se) | 457 | 17.9% | 20.1% (16.7 to 24.0) | 11.8% |
| Czechia (.cz) | 461 | 12.6% | 17.1% (14.0 to 20.8) | 6.5% |
| Austria (.at) | 412 | 13.3% | 16.5% (13.2 to 20.4) | 6.6% |
| Belgium (.be) | 418 | 14.1% | 16.5% (13.3 to 20.4) | 8.9% |
| Ireland (.ie) | 425 | 13.4% | 15.8% (12.6 to 19.5) | 6.6% |
| Portugal (.pt) | 419 | 13.1% | 15.8% (12.6 to 19.5) | 3.6% |
| Slovakia (.sk) | 426 | 10.6% | 15.7% (12.6 to 19.5) | 4.2% |
| Greece (.gr) | 420 | 10.7% | 15.2% (12.1 to 19.0) | 3.6% |
| Latvia (.lv) | 392 | 13.0% | 14.8% (11.6 to 18.7) | 4.3% |
| Luxembourg (.lu) | 156 | 14.1% | 14.7% (10.0 to 21.2) | 8.3% |
| Estonia (.ee) | 441 | 10.7% | 14.3% (11.3 to 17.9) | 4.1% |
| Lithuania (.lt) | 412 | 10.9% | 14.3% (11.3 to 18.0) | 4.9% |
| Croatia (.hr) | 429 | 10.7% | 13.3% (10.4 to 16.8) | 4.0% |
| Romania (.ro) | 430 | 9.1% | 12.8% (10.0 to 16.3) | 2.3% |
| Hungary (.hu) | 440 | 10.9% | 12.5% (9.7 to 15.9) | 4.5% |
| Bulgaria (.bg) | 428 | 7.9% | 11.9% (9.2 to 15.3) | 3.3% |
| Slovenia (.si) | 410 | 9.3% | 11.2% (8.5 to 14.6) | 3.9% |
Sorted by the fourth column. Answer-engine search crawler blocked most: Germany 16.6%, Italy 16.5%, Netherlands 15.6%; least: Romania 2.3%. Luxembourg has 156 sites read; its interval is wide. Download by country (CSV)
By CMS
| CMS (home-page signature) | Sites read | Block GPTBot | Block at least one of the four |
|---|---|---|---|
| No signature matched | 5,635 | 16.1% | 19.9% |
| WordPress | 1,667 | 8.9% | 10.1% |
| Home page not fetched | 1,425 | 23.9% | 27.5% |
| Next.js | 584 | 18.7% | 20.7% |
| Drupal | 533 | 8.3% | 9.4% |
| Magento | 207 | 7.7% | 10.1% |
| PrestaShop | 140 | 9.3% | 17.1% |
| TYPO3 | 132 | 6.8% | 7.6% |
| Joomla | 61 | 14.8% | 14.8% |
Only CMSs with at least 50 sites are shown; Shopify, Wix, Squarespace and Webflow have fewer and are not reported. The CMS is a signature in the home page, not a verified platform, and "No signature matched" is the largest group. Download by CMS (CSV)
Method and limits
- Question: of the most popular sites under each EU country-code domain, which AI crawlers does robots.txt ask to stay out?
- Sample: the top 600 domains by Tranco rank under each of 25 EU ccTLDs (Luxembourg has 250), from Tranco list 56Z8N (created 22 Oct 2026; tranco-list.eu/list/56Z8N/1000000). 14,650 domains probed; robots.txt read on 10,426 (71%).
- Fetch: one GET of
/robots.txt(https, then www, then http) and one GET of the home page for a CMS signature. User-AgentCitedBy-research/1.0 (+https://citedby.eu; robots.txt study, one request per site). No login, no cookies, no retry after a refusal, no browser identity. - Reading: RFC 9309: the group naming the crawler wins, else the
*group; longest rule wins; Allow wins a tie. "Blocks" = the crawler may not fetch/. 16 crawlers tracked (14 AI-related, Googlebot and Bingbot as controls). - Share base: the 10,426 sites whose robots.txt was read. The other 4,224 are coverage, not "allows": 1,729 unreachable, 1,140 answered HTTP 403 (to our research UA), 713 had no robots.txt (404), 436 returned an HTML page instead, 206 other.
- Precision check (23 Oct): 20 sites drawn at random (8 verdict "blocked", 4 "allowed", 8 "partial" for GPTBot) re-fetched and the GPTBot /
*lines read by eye: 0 contradictions. Two stacked-group cases (tv2.dk, veikkaus.fi) were not read to the end of the group. - Parser test: 7 unit tests, including the
Disallow: /?case that python's robotparser reads as/(our lesson from earlier work). - Limits: robots.txt states a request, not what a firewall enforces: 7.8% of probes got HTTP 403 for our research user-agent, which may be a bot block but is not counted as one. ccTLD is not the operator's country (a German company can use .com; a .de site can be foreign-owned). Tranco ranks popularity, so the sample is the popular end of each ccTLD, not a random draw of all sites. Sites whose robots.txt we could not read (29%) may differ from those we could: of the readable sites, those whose home page we could not fetch block more often (GPTBot 23.9% vs 15.3% overall), so the unread 29% probably lean toward more blocking, not less (an inference, not a measurement). The first 2,900 probes used a 12 s timeout and 40 threads, the rest 8 s and 120 threads; this can only move a site between "read" and "unreachable". CMS is a home-page signature, not a verified platform ("No signature matched" is the largest group).
- Raw data: one row per domain with the verdict for each crawler, public robots.txt data only: ai-access-index-2026-10.csv (2 MB). Licence: CC BY 4.0, cite as "CitedBy AI-Access Index, run 1, October 2026".
Run history
The crawl is repeated each month on the newest Tranco list. This page is run 1 (read 23 Oct 2026) and stays at /data/ai-access-index-2026-10; /data/ai-access-index always shows the latest run.
What does your robots.txt say to AI crawlers?
We crawl your site up to your plan's page limit and give you the exact fix for each issue. From €9, 14-day money-back guarantee.
Checked on 23 Oct 2026 · Source: RFC 9309 (robots.txt) · Tranco list 56Z8N · Method: How CitedBy checks · Change log: What changed