AI crawler access API

One request tells you which AI crawlers can read a site: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot. For each one you get what robots.txt says and what the server answers it. It is the result the free AI crawler check shows, as JSON. No key, no account, CORS open.

Request

One GET request with one parameter:

GET https://citedby.eu/api/v1/ai-access?url=example.com

url: A domain or a full address, for example example.com or https://example.com/shop. Only the site is checked: the path, the port and www. are dropped.

Example

The check of theguardian.com on 24 October 2026: a public site, not a CitedBy customer, in the result as the API returned it. Run it on another day and the answer can differ.

curl "https://citedby.eu/api/v1/ai-access?url=theguardian.com"

{"api_version": "1", "site": "theguardian.com", "checked_at": "2026-10-24T17:27:07Z", "robots_txt": {"state": "found", "status": 200, "blocks_all_crawlers": false}, "summary": {"total": 8, "allowed": 6, "blocked": ["PerplexityBot", "ClaudeBot"]},
  "crawlers": [
    {"id": "GPTBot", "role": "training", "robots_txt": "allowed", "robots_txt_rule": null, "server": "ok", "access": "allowed"},
    {"id": "OAI-SearchBot", "role": "search", "robots_txt": "allowed", "robots_txt_rule": null, "server": "ok", "access": "allowed"},
    {"id": "ChatGPT-User", "role": "user_fetch", "robots_txt": "allowed", "robots_txt_rule": null, "server": "ok", "access": "allowed"},
    {"id": "PerplexityBot", "role": "search", "robots_txt": "disallowed", "robots_txt_rule": "Disallow: /", "server": "blocked", "access": "blocked_by_robots_txt_and_server"},
    {"id": "ClaudeBot", "role": "training", "robots_txt": "disallowed", "robots_txt_rule": "Disallow: /", "server": "blocked", "access": "blocked_by_robots_txt_and_server"},
    {"id": "Google-Extended", "role": "training", "robots_txt": "allowed", "robots_txt_rule": null, "server": "not_tested", "access": "allowed"},
    {"id": "Bingbot", "role": "search", "robots_txt": "allowed", "robots_txt_rule": null, "server": "ok", "access": "allowed"},
    {"id": "Googlebot", "role": "search", "robots_txt": "allowed", "robots_txt_rule": null, "server": "ok", "access": "allowed"}
  ],
  "report_url": "https://citedby.eu/tools/ai-crawler-check/theguardian.com?utm_source=api&utm_medium=api&utm_campaign=ai_access_api", "audit_url": "https://citedby.eu/?url=theguardian.com&source=api&utm_source=api&utm_medium=api&utm_campaign=ai_access_api", "attribution": {"text": "AI crawler access by CitedBy", "url": "https://citedby.eu/ai-access-api"}, "docs": "https://citedby.eu/ai-access-api"}

What the response holds

Every field below is in every successful response.

site: The domain that was checked.
checked_at: When CitedBy ran the check, in UTC. A site checked in the last 6 hours returns the stored result, so this can be earlier than your request.
robots_txt.state: found, not_found, unreachable or server_error. A server error on robots.txt means a crawler has to assume it may not crawl (RFC 9309), so every crawler then shows as blocked.
robots_txt.status: The HTTP status robots.txt answered with.
robots_txt.blocks_all_crawlers: true when the rules for all crawlers (*) close the whole site.
summary: How many of the 8 crawlers can read the site (allowed), and the names of those that cannot (blocked).
crawlers[].id: The crawler's name.
crawlers[].role: training (collects text to train models), search (can cite your pages in answers) or user_fetch (opens a page when a person asks about it).
crawlers[].robots_txt: allowed, disallowed or not_read (robots.txt could not be read).
crawlers[].robots_txt_rule: The robots.txt rule that closes the site to this crawler, or null.
crawlers[].server: ok, blocked (the server answers this crawler's user-agent with a refusal or a challenge page), down (an error) or not_tested. Google-Extended is a robots.txt token only, so it is never tested on the server.
crawlers[].access: The verdict for this crawler, one of the values below.
report_url: The full result page of the free check for this site.
audit_url: Where to audit the whole site.
attribution: The text and link to show when you display the result (see the terms).

The access values

allowed means robots.txt does not close the site to the crawler and the server answered it. The others name what stands in the way:

allowed: the crawler can read the site.
blocked_by_robots_txt: robots.txt closes the site to this crawler.
blocked_by_server: the server turns this crawler's user-agent away or shows it a challenge page.
blocked_by_robots_txt_and_server: both.
server_error: the server answered this crawler with an error.

Blocking a training crawler can be a choice. The API reports it as blocked and does not call it a problem.

Errors

A failure is an error, never a guess. The body is JSON: {"error": {"code", "message"}}.

missing_url: 400: the url parameter is missing.
bad_url: 400: not a public site address (a private address, a bad name, more than 2,048 characters).
site_unreachable: 422: the site did not answer our crawler, so access cannot be judged. Nothing is guessed: there is no all-clear.
rate_limited: 429: over the limits below. Retry-After says how many seconds to wait.
busy: 503: CitedBy is running its maximum number of checks. Retry-After is 60 seconds.
unavailable: 502 or 503: the checker did not answer. Try again in a minute.

A slow site returns 202 with {"status": "pending"}: the check keeps running and its result is stored. Ask again after the Retry-After seconds.

Limits

30 requests per hour for each IP address.
At most 12 sites per hour that were not already checked in the last 6 hours. A site that was checked in that time is answered from the stored result and does not use this allowance.
No key and no sign-up: the limits are the only gate. The result is cached for 6 hours.

What it does not return

No grade and no score. The API returns what the free check shows for each crawler.
No page-by-page crawl, no exact fix per page and no llms.txt check: those are in the audit.
It tests the home page with each crawler's user-agent string, from our own address. A firewall that checks the vendors' IP ranges can treat the real crawlers differently.
A robots.txt that redirects to another domain is not followed: it shows as not_found with the redirect's status.

OpenAPI

The spec is at https://citedby.eu/openapi.json. Load it in Postman, Insomnia or a code generator.

Terms of use

Free to use, also in your own product, with no key or account.
If you show our results to other people (a page, a report, a README, a tool), link to CitedBy: https://citedby.eu/ai-access-api or the report_url of the site. The attribution field carries the text and the link.
Do not present the result as your own check, and do not send bursts: the limits apply to each address, and we can change or end them. There is no uptime promise.
We count the calls, as a number per day. The address you call from is used for the limits and is kept in memory, not in a database. More in Privacy.

Same check, other ways

English · Français · Nederlands · Deutsch · Español

Need more than crawler access? A CitedBy audit crawls your site, up to your plan's page limit, and gives you the exact fix. See the plans.