Which AI crawlers do your clients' sites block?
Paste up to 25 domains, one per line. For each site we read its robots.txt the way a crawler does and show which of 14 AI crawlers it lets in, in one table you can download as CSV. Free, no account.
What you get
- One row per site: whether robots.txt was found, which training crawlers it blocks (GPTBot, ClaudeBot, Google-Extended, CCBot) and which answer-engine search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
- The share of your list that blocks each kind, beside the same share across popular EU sites in our weekly AI-Access Index.
- A CSV in the Index's columns, one row per site, for your own report.
How the list is read
Each robots.txt is requested at https, then https with www, then http, and read under RFC 9309: the group that names the crawler wins, else the * group; the longest matching rule wins. "Blocks" means the crawler may not fetch the home page. This is what each site asks; the single-site check also tests what its server answers to each crawler.
Who it is for
Agencies and consultants who need to tell a client in one line whether GPTBot, ClaudeBot or PerplexityBot can read their site, and which clients to ring first.