# CitedBy: full text of the key pages > The plain text of the English pages listed in /llms.txt, in one file. CitedBy crawls your site up to your plan's page limit and gives a fix for each issue, with the code to paste where there is one. The only domain is citedby.eu. ## CitedBy: AI search audit with the fix for every page Source: https://citedby.eu/ See what your site sends to Google and AI crawlers. Paste your address. CitedBy reads your site the way a crawler does and tells you what to fix, page by page. Free, no account, no card. We check up to 10 pages and show every issue with its fix. This is what you get: a real report citedby.eu/sample What the crawl found 9 issues, each with its fix 0 Errors 0 Warnings 9 Notices 6,496 pages crawled Fix these 3 first Many sitemap URLs come from one repetitive template 122 sitemap URLs Many pages you link to ask search engines not to index them about 2,294 linked pages Many sitemap URLs are dated the same day 310 sitemap URLs AI crawler access robots.txt, plus one live request per crawler Crawler robots.txt Live test PerplexityBot allowed 200 OAI-SearchBot allowed 200 Claude-SearchBot allowed 200 GPTBot allowed 200 ClaudeBot allowed 200 One issue, with its fix Notice Article has no named author 121 pages Page: /ai-access-api · author is the organization (CitedBy editorial team) (one of 121 pages) How to fix it Name the person who wrote or reviewed it: add a byline and an author of type Person (name, and a profile or sameAs link) to the Article JSON-LD. Our crawl of our own site, citedby.eu, 1 Nov 2026, 6,496 pages. Open the full report. For the whole site: full audit, €9 once (1 site, up to 500 pages, a fix for each issue, with code to paste where there is one). 14-day money-back guarantee. Every problem comes with its fix Free one-page checkers Test the page you paste. Your other pages stay unchecked. Crawlers you install Give you a spreadsheet of problems. You work out the fixes. CitedBy Reads every page up to your plan limit and gives you a fix for each issue, with the tag, robots.txt line or schema block to paste where there is one. A problem repeated on 100 pages is one fix with the page count shown, not 100 errors. Which tool fits which job, with prices read 28 October 2026. What we check, page by page and across the sitemap Indexing risk comes first: a template flood came just before our own site lost 93% of its impressions. Indexing risk Template floods, same-day sitemap dates, noindex URLs in the sitemap, near-duplicate pages. Links, speed and canonicals Broken links, redirect chains, canonicals, slow and heavy pages, orphan pages. Titles, headings and schema Titles, descriptions, H1s and structured data that Google's rich-result check rejects. Readable content Pages that arrive empty until JavaScript runs, thin pages, noindex tags. Built with Lovable? AI crawler access robots.txt and firewall blocks for OAI-SearchBot (ChatGPT search), GPTBot (OpenAI training) and others, with a live request each. Answer quality FAQ, pricing, comparison and about pages, and a page that answers each question buyers ask. Every check, with what it reads. We report llms.txt as an optional notice, not an error: why. Google-Extended is a robots.txt setting with no crawler of its own, so we read it from robots.txt only. What happened to one of our young sites, and what CitedBy's crawl of 28 Oct says ×300 About 300 times more sitemap URLs from 17 to 22 Oct; by 27 Oct, 67% came from one template. −93% Google impressions from 25 to 26 Oct, −96% by 27 Oct. Google does not publish its reasons. C- 72 CitedBy's crawl of 28 Oct, against A 96 on 26 Oct, before the new checks: 59% of sitemap URLs come from one template. Sources: daily figures (CSV), our Search Console, and the full write-up with every figure. Did your sitemap, templates or robots.txt change this month? The €9 audit runs the same checks on your site. Check my site free From URL to fix list 1. Enter your URL Start with the free check above, no account needed. The full report is the €9 audit or a plan, because the full crawl costs us money. 2. We crawl up to your plan's page limit Up to 500 pages on a Single audit, up to 6,500 on Agency. You watch the count rise live. 3. Read the report What to fix first, every issue ranked by severity, and a page-by-page view. 4. Make the fixes, then recrawl Each issue has its fix. Recrawl weekly on Starter, daily on Pro and Agency, and watch the score move. citedby.eu/sample Our crawl of our own site, citedby.eu, 1 Nov 2026, 6,496 pages. Open the full report. Pick how often you audit Every plan has the full report with a fix for each issue (code to paste where there is one), and a 14-day money-back guarantee. Single audit €9 one-off One site, one report 1 site, up to 500 pages A fix for each issue, code where there is one PDF report Report stays in your account €9 once Run one audit for €9 Starter €19 a month Keep one site healthy 1 site, up to 1,000 pages per crawl Weekly re-crawl and score trend Email alerts on new issues PDF report €19 a month per site €0 today, then €19 a month. Card needed. Start 7-day free trial Recommended Pro €49 a month For freelancers and teams with a few sites 5 sites, up to 5,000 pages per site Daily re-crawl and score trend Email alerts on new issues PDF reports €9.80 a month per site €0 today, then €49 a month. Card needed. Start 7-day free trial Agency €99 a month Client sites under your name 25 sites, up to 6,500 pages per crawl Daily re-crawl and score trend Email alerts on new issues White-label PDF with your logo Your brand on shared report links €3.96 a month per site €0 today, then €99 a month. Card needed. Start 7-day free trial Yearly billing gives you two months free: Starter €190, Pro €490, Agency €990 a year. Yearly plans have no trial and are charged when you start. Prices include VAT. The trial covers 1 site, up to 500 pages and 2 audits; your plan's full allowance starts with the first payment. One trial per email address and card. Cancel any time in your account. Pay by card through Stripe, we never see your card. 14-day money-back guarantee, on every plan. If CitedBy was not worth it, write to hello@citedby.eu within 14 days of paying, including after the first charge of a trial, and we refund the full amount. You do not need a reason. Refund terms The date and amount, up front Before you confirm a trial, you see the first charge: the date and the euros. A card is needed to start, and €0 is charged today. A reminder 3 days before We email you 3 days before the first charge, with the date, the amount and a link to cancel. Cancel in your account Any time, from your account page. Cancel before the first charge and you pay nothing. After it, 14 days money back. Questions Why did my Google impressions drop overnight? Open Search Console and read Manual actions, Security issues and the Page indexing report first. Then list what your site changed in the days before: a burst of new pages from one template, sitemap dates that all moved, changed titles, canonicals or noindex tags. A crawl shows those patterns in one pass. On one of our own young sites, impressions fell from 612 to 44 in a day after its sitemap grew from 6 to 1,806 URLs in five days ( numbers and reports above ). Google does not say why a site loses impressions, so a cause stays likely, not proven. What does “Crawled, currently not indexed” mean? Google fetched the page and has not put it in its index. Google's help page says this is common and does not necessarily mean something needs fixing. In our own crawls the usual suspects are text that repeats a sibling page, very little text and a page nothing links to. CitedBy checks each of these on your crawl and shows the share of sitemap pages that will probably not be indexed, labelled as an estimate. Connect Search Console and it adds Google's own counts beside the estimate. Can ChatGPT and Bing see my JavaScript site? Only text that is in the HTML your server sends is sure to reach every crawler. We read each page the way a crawler that does not run JavaScript does, and report a page that arrives empty until scripts run, with the fix. Whether a given bot runs scripts differs by bot and changes, so we do not guess: the free AI crawler check sends one real request per crawler and shows what each one got back. Why pay for an audit when free checkers exist? Free checkers test one page, and crawlers you install hand you a list of problems. CitedBy checks every page up to your plan limit and writes the fix for each issue, with the code to paste where there is one, so you skip the research. If it is not worth €9 to you, the 14-day guarantee refunds it. What is an AI search audit and why do I need one? An AI search audit checks whether AI assistants and search engines can reach and read your pages: crawler access in robots.txt, text readable without JavaScript, structured data, titles and internal links. You need one because an assistant cannot use text it could not read. For each issue the report gives the fix, with the robots.txt line or JSON-LD block to paste where there is one. Whether an assistant then names or links your site also depends on what other sites say about you. To see that today, ask the assistants your buyers' questions with web search on. How do I check if AI assistants can read my website? Paste your address into the free check on this page, or use the free AI crawler checker at citedby.eu/tools/ai-crawler-check. It shows whether robots.txt and your server let GPTBot, ClaudeBot, PerplexityBot and Googlebot in, with the robots.txt lines to change, or where to look on your server. How much does a website audit for AI visibility cost? A single audit is €9 once, for 1 site and up to 500 pages. Starter is €19 a month (up to 1,000 pages), Pro €49 a month (5 sites, up to 5,000 pages each) and Agency €99 a month (25 sites). Every paid plan has a 14-day money-back guarantee. What is the difference between SEO and AI search readiness? SEO is about ranking in a list of links. AI search readiness is whether an assistant can fetch your page and read its text without running JavaScript, and whether your site answers the questions buyers ask. They overlap on crawlability, titles and structured data, and differ on AI crawler access, which robots.txt sets for each crawler (OAI-SearchBot, GPTBot, ClaudeBot). More in GEO vs AEO vs SEO. Can I fix AI crawler access issues myself? Yes. The audit checks two places where a crawler can be kept out: your robots.txt and your host or CDN, such as its firewall or bot protection. Each robots.txt issue in the audit comes with the lines to paste, and our pages for Shopify, Squarespace, Wix, Webflow, WordPress.com and Cloudflare show where each host or firewall setting sits. How do I know if ChatGPT can find my site? ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page when someone asks about it. If robots.txt or your firewall blocks them, ChatGPT cannot read your pages, and the free AI crawler check tests both. Whether it then names you in an answer is the assistant's own choice. What structured data do AI assistants need? No major assistant publishes a required list. JSON-LD tells them what a page is: Organization on the home page, Product on product pages, Article on posts and FAQPage where a page answers questions. The audit flags pages with none and writes the JSON-LD to paste where it can build one from the page. How long does a website audit take? Our crawler starts at most about 8 requests a second on your site and slows down when your server answers 429 or 503. The crawl stops after 15 minutes at the latest, then the checks and the analysis run. The report opens in your account when they finish. How does the 7-day free trial work? Monthly Starter, Pro and Agency start with 7 days free. Yearly plans have no trial and are charged when you start. You enter a card, we charge €0 today, and the Stripe page shows the date and amount of the first charge before you confirm. The trial covers 1 site, up to 500 pages and 2 audits, with the full report; your plan's full allowance starts with the first payment. We email you 3 days before the first charge with a cancel link. Cancel in your account, and if you cancel before the first charge you pay nothing. One trial per email address and card. Why do you need my card for a trial? A crawl costs us real money, and a card keeps the trial to one per person. The charge date and amount are shown up front, you get a reminder 3 days before, and after the first charge the 14-day money-back guarantee still applies. If you want no subscription at all, the €9 audit has none. What do I get for €9? One audit of one site, up to 500 pages: a short list of what to fix first, every issue ranked by severity with the pages it affects, a page-by-page view, a fix for each issue with the code to paste where there is one, and a PDF. Your report stays in your account. What does the audit read? The HTML each page serves, plus robots.txt, your sitemap and llms.txt. We read public pages only. We do not log in. Each page is read the way a crawler that does not run JavaScript reads it, so a page that arrives empty until JavaScript runs is reported as exactly that. On up to 20 pages of a site audit we also open the page in a browser and count what it shows once its scripts have run; the free check counts only what a crawler gets. A check we could not read is shown as "not measured", never as a pass. What if you cannot crawl my site? If your firewall or host blocks CitedBot, the report says so and shows how to allow it. Change the setting and write to us, and we look at it with you. If it is not worth it to you, the 14-day money-back guarantee covers it, no reason needed. Which flags can I ignore? Every issue has a severity, so you start with the high ones. Checks that are optional say so: no major assistant is confirmed to read an llms.txt file, and Ahrefs found 97% of valid files got no requests in May 2026 (137,210 sites). A check we could not read is shown as "not measured", never as a failure. Will fixing the issues make ChatGPT mention my site? The audit finds what blocks or weakens your pages for AI search and gives you the fix. That is the part you control; which sites an assistant names is its own choice. Can I cancel? Yes, in your account, any time. You keep access until the end of the period you paid for. Inside 14 days of paying you can also ask for the full amount back. Run the free check on your site From €9. 14-day money-back guarantee. Also free, one domain each: AI crawler check · llms.txt checker · llms.txt generator · Findable by Google and Bing · Client list check · All free tools We read public pages only and never log in Our crawler is CitedBot, documented at citedby.eu/bot Pay by card through Stripe CitedBy is run from Spain. Legal notice ## About CitedBy: website audit for AI search | CitedBy Source: https://citedby.eu/about About CitedBy CitedBy (citedby.eu) is a website audit for AI search. You enter a URL, we crawl your site up to your plan's page limit, check whether AI search crawlers can reach and read it, and give you a graded report with the exact fix for each issue. Which CitedBy is this This is citedby.eu. It is not citedby.co or citedby.ai, which are different companies, and it is not an academic citation tool : it does not format, count or look up scholarly references. The only domain we use is citedby.eu. What we do Crawl your site from one URL, up to your plan's page limit, and score every page crawled for AI-crawler access, readable content, llms.txt, structured data, titles and headings and internal links. Give the fix for each issue, copy-paste where possible. Re-crawl on a schedule so you can see the score move; white-label PDF reports and multi-site monitoring for agencies. Keep a dated list of changes the engines published, with what you can do and whether we check it. What we sell We check what assistants and search engines can read on your site and show the fix for each finding. We sell audits and plans, never reviews, links or mentions. How each number is produced is on the methodology page. How we earn money From audits and plans, listed on the pricing page, with a 14-day money-back guarantee ( refund terms ). We take no commission from anything we recommend. Editorial standards Who writes. The guides, studies and reports on this site are published by CitedBy under its name, not an individual's. Where claims come from. Each article is written from the public documents it cites, listed under Sources at the end. Numbers we measured come from our own crawls and the data pages. When a number comes from someone else's study, we name that study in the text and under Sources. What "Page updated" means. The date the article's text, sources or numbers last changed. Corrections. If you find an error, write to hello@citedby.eu with the page address. We fix it and update that date. Method. How each number is produced is on the methodology page. Who runs it CitedBy is operated by an individual in Spain. The operator's name, postal address and contact details are in the legal notice, as Spanish law (LSSI-CE art. 10) requires. The reports and guides on this site are published by CitedBy. Contact hello@citedby.eu. We reply within 2 working days. ## How a CitedBy audit works | CitedBy Source: https://citedby.eu/methodology How a CitedBy audit works Enter a URL. CitedBy crawls the site, checks every page it finds and gives you a graded report with the exact fix for each issue. This page says what is checked, how the score is built and what a score tells you. 1. The crawl We start from your URL, read robots.txt and the sitemap, and follow internal links to find pages and sub-pages. Each plan has a page cap; the single audit covers up to 500 pages. We identify ourselves with a CitedBy user agent and stay on your domain. Every request is made from the public internet, logged out, like any crawler would. 2. What is checked Crawler access. robots.txt rules for GPTBot, ClaudeBot and PerplexityBot, the status code your home page returns to each of those user agents (one request per bot), and signs of a firewall or CDN block. Readable HTML. The text a crawler receives without running JavaScript, compared with the size of the page. Pages that arrive empty are flagged. On up to 20 pages (the home page, then the page with the least text of each template, then other pages with little text) we also open the page in a browser and count the words it shows after its scripts run. The report then gives the median of each count over those pages and lists the pages where the two differ most; without a browser count it lists the pages where a crawler sees the fewest words. A page the browser could not open has no browser count, never a 0. The free check counts only what a crawler gets. llms.txt and sitemap. Whether the files exist and parse. No major assistant is confirmed to read llms.txt, so it is marked optional and never drags the score down on its own. Structured data. JSON-LD for Organization, Product, FAQ and Article, checked for validity and for fields a buyer would ask about, such as price. Titles, headings and links. Missing or duplicate titles and descriptions, H1s, heading order, canonical tags, hreflang, broken internal links. Answer quality. Checked once for the whole site, not page by page: whether the site has FAQ, pricing, comparison and about pages, and whether some page answers each question buyers ask assistants about your category. We report a missing pricing, comparison or about page, or an unanswered question, only when the crawl read every page it found, never from a crawl stopped at the plan's page cap or time limit. Before we call a question unanswered we search the title, headings and opening text of every page we crawled. 3. How the score is built Each check passes or raises an issue per page. Issues are grouped by severity (Error, Warning, Notice) with the number of pages each one affects. Sections get a letter grade from A+ to F and the site gets one overall grade. Open any issue to see every affected URL with the value we measured on it (for example the word count or the status code) and the date of the crawl, so you can check any count against your own site. Could not verify. If a page refuses our crawler (HTTP 401, 403 or 429), shows a firewall challenge, is blocked by your robots.txt or does not answer, we cannot tell what is on it. We list it as “could not verify” with the reason, skip its checks and leave its issues out of the score, so a firewall does not add issues to your list. Allow our crawler and re-crawl to have those pages checked. Grade ceiling. A grade says something about the whole site only when most of its pages were read. When 1 in 10 or more of the crawled pages returns an error (HTTP 4xx or 5xx) or could not be verified, the overall grade cannot go above B (83), and the report says so and shows the score without that limit. Fix the errors or let our crawler in and re-crawl to lift it. The free check reads up to 10 pages, so it shows the number of issues found in that sample and no grade. 4. The fixes Where a fix can be pasted, we give it: the robots.txt lines, a JSON-LD block, an llms.txt file. Re-crawl after you change something and the score moves with it; scheduled re-crawls show the trend. 5. What a score tells you A good score means assistants and search engines can reach and read your pages: it reports what we found on access and readability. Whether an assistant names or links your site also depends on what other sites say about you. Whether Google has indexed your pages is a separate number, Findability, which we show only when Search Console is connected and otherwise mark as not measured. We do not report a "visibility percentage" from a sample of assistant answers. Anything we have not measured is shown as not measured, never as 0. Some checks are judgements (does any page answer a buyer question). They are marked as such and carry less weight than the access and readability checks, which are exact. We do not score whether a page opens with its answer or is easy to quote: that check is switched off until it passes our accuracy test. We do not use private or consumer chat interfaces of any vendor. 6. How often we are wrong Every finding in a report has a "This is wrong" button. A mark is stored with what we showed and goes to our weekly review of the rule. When you press "Verify this fix" and the page now passes, that finding counts as one we got right. Rules precision is 1 minus the findings marked wrong divided by the findings reviewed (marked wrong plus fixes verified), over the last 30 days, in customer audits only. We publish a rate once 10 findings are reviewed; below that we show the counts. The same figures per check are at /api/precision. The live counts are at /api/precision. Last measured on our own sites (26 October 2026): 3 of 66 findings were wrong. Reading 1, the current engine replayed on crawls of three of our sites, saved on 23 October 2026: 0 of 36 wrong (partly right: 2; not judged: 12). Reading 2, the latest live audits of those three sites and resaleiq.dev, each finding checked against the live page: 3 of 30 wrong (11 more were right when the audit ran and the page has changed since). Every finding with its verdict: the table and the two CSVs. Changes to the method Last updated 8 November 2026. When a check is added or changed, this page changes with it. Page checks added since 19 Oct 2026, with the day each went live: checks we added. Every check has its own page, with the fix and how common it is: check library. Who operates CitedBy and which CitedBy this is: About CitedBy. Questions about a result: hello@citedby.eu. ## CitedBy data: who AI assistants link for shop questions Source: https://citedby.eu/data Who AI assistants link when buyers ask where to shop Dated counts from our own runs of Claude and ChatGPT with web search on: which shop sites appear in the answers to buyer questions, by niche and country. Every page shows the questions, the method and the raw counts. Each page names the one assistant it counted. Studies EU AI-Access Index: AI Crawler Permissions Across European Websites: robots.txt of 14,650 popular EU domains, stable CSV and JSON, time series by country and CMS, weekly EU AI-Access reference: AI crawler rules and llms.txt of popular EU sites, one page per country and CMS, weekly AI citation map: which sources Claude cites for buyers' questions, 4 niches x 4 languages, monthly EU llms.txt Index: how many of 10,223 popular EU sites publish an llms.txt Launch crawl 2026-10: Show HN and Lovable sites read as a crawler reads them Finding audit: how often CitedBy findings are wrong, read one by one on four of our own sites Check library: every check of the audit with its fix and how common it is, in five languages Checks we added: the page checks added since 19 Oct, by the version and day each went live Is citedby.eu findable? Google and Bing index counts and AI crawler fetches on our own site (EN) 48 Claude answers to 24 buyer questions in 3 EU niches: who gets linked 48 ChatGPT answers to 24 buyer questions in 3 EU niches: who gets linked By niche and country Quelles marques Claude cite pour acheter un matelas en ligne en France (FR, Claude, 16 answers, 19 Oct 2026) Quelles marques Claude cite pour acheter des compléments alimentaires en ligne en France (FR, Claude, 16 answers, 19 Oct 2026) Quelles marques Claude cite pour des compléments beauté et du collagène en France (FR, Claude, 16 answers, 19 Oct 2026) Want a report on your own site? CitedBy crawls your site up to your plan's page limit and gives you the exact fix for each issue. From €9, with a 14-day money-back guarantee. Audit my site, from €9 ## CitedBot, our site crawler. CitedBy Source: https://citedby.eu/bot CitedBot CitedBot is the crawler behind CitedBy site audits: it crawls your pages, analyses them and reports what to fix. Its User-Agent header is: Mozilla/5.0 (compatible; CitedBot/1.0; +https://citedby.eu/bot) When it visits Only when a site owner or a visitor asks CitedBy to audit that site, or asks their AI tool to check one page through the CitedBy MCP server (one page, the same rules). It does not crawl on its own, and it does not build an index of the web. How it behaves It reads your robots.txt first and obeys it. It follows the group User-agent: CitedBot and, if there is none, the group User-agent: *. When it crawls, it spaces its requests to your site: at most about 8 a second, and slower, up to 2 seconds apart, when your server answers 429 or 503. It stops at a page limit set by the plan of the audit: 500, 1,000, 5,000 or 6,500 pages. It only reads public pages. It does not log in or submit forms. On up to 20 pages per site audit (the home page, then the page with the least text of each template, then other pages with little text), it also opens the page in a browser and runs its scripts, as a visitor's browser would. These pages open one at a time, with your Crawl-delay between them (up to 5 seconds). Inside a page, the browser loads scripts and data as fast as a visitor's would, and checks each address against robots.txt before requesting it. The free check never opens a browser. Block it Add this to your robots.txt and it will not fetch any page: User-agent: CitedBot Disallow: / To keep it out of one folder only, use Disallow: /folder/ instead. Contact Questions or a request to stop crawling: hello@citedby.eu. We reply within 2 working days. ## robots.txt for AI bots: allow or block ChatGPT, Claude. CitedBy Source: https://citedby.eu/bots robots.txt for AI crawlers: allow or block ChatGPT, Claude and Perplexity Copy-paste robots.txt blocks to allow or block ChatGPT, Claude and Perplexity come first, then one page per crawler, with the operator, the robots.txt token, the operator's own sentence, what blocking changes and how to verify a request, each read from the operator's own page on the date shown. By CitedBy · Page updated 6 November 2026. method Copy-paste robots.txt blocks Three blocks to copy into your robots.txt. Crawlers whose operators say they may be used to train models: GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, Amazonbot. Search and answer crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, Meta-WebIndexer. This is a choice for your business, and a Disallow only covers well-behaved bots. Allow all AI crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Amzn-SearchBot User-agent: Meta-WebIndexer Allow: / Allow AI search, block training User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot Disallow: / User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Amzn-SearchBot User-agent: Meta-WebIndexer Allow: / Block all AI crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Amzn-SearchBot User-agent: Meta-WebIndexer Disallow: / Also stop Gemini using your pages Google-Extended turns off both training and grounding in Gemini Apps, so Gemini's live answers stop using your pages. Add it only if you want that. User-agent: Google-Extended Disallow: / One shared line for AI training On 30 October 2026 Google, OpenAI and Microsoft said they plan to read one shared robots.txt line for AI training (Google and Microsoft in 2027, OpenAI in the coming months). Google plans to read an opt-out there as a Google-Extended block, grounding in Gemini included, unless you name Google-Extended. The second group keeps your pages eligible for Gemini's grounded answers and also allows Gemini training; leave it out if you want neither. With no such line, no preference is expressed, and a User-agent: * group does not count for it. User-agent: __usage_ai-training Disallow: / User-agent: Google-Extended Allow: / An AI crawler is a bot that fetches pages for an assistant such as ChatGPT, Claude, Perplexity or Gemini, for a search index or for model training. Each page names the operator, the robots.txt token, what blocking changes and how to verify a request, so you can decide per bot whether to allow it. Jump to a job: Search (18) Training (3) User-triggered fetch (11) Several jobs (6) Control token (2) All 40 AI and search crawlers on this page set, with the operator, the job and whether the operator says it follows robots.txt. Crawler Operator Job Follows robots.txt Search (18) Amzn-SearchBot Amazon Search Yes Baiduspider Baidu Search Yes Bingbot Microsoft Search crawler Yes Claude-SearchBot Anthropic Search Yes Diffbot Diffbot Search (general web crawl) Yes DuckDuckBot DuckDuckGo Search Yes ExaSearchBot Exa Search and retrieval index Yes Googlebot Google Search (also Images, Video, News, Discover) Yes ImagesiftBot Hive (ImageSift) Image index for similar-image search Yes Kagibot Kagi Search Yes LinkupBot Linkup Search index for AI web search Yes Meta-WebIndexer Meta Search Yes MojeekBot Mojeek Search Yes OAI-SearchBot OpenAI Search Yes PerplexityBot Perplexity Search Yes Storebot-Google Google Search (Google Shopping surfaces) Yes YandexBot Yandex Search (main indexing robot) Yes YouBot You.com Search Yes Training (3) ClaudeBot Anthropic Training Yes GPTBot OpenAI Training Yes ICC-Crawler NICT (Universal Communication Research Institute) Research data collection (includes AI research and development) Yes User-triggered fetch (11) Amzn-User Amazon User-triggered fetch Not always ChatGPT-User OpenAI User-triggered fetch Not always Claude-User Anthropic User-triggered fetch Yes DuckAssistBot DuckDuckGo Real-time fetch for AI-assisted answers Yes ExaUser Exa User-triggered fetch Not always Google-Agent Google User-triggered fetch (agents acting for a user) Generally not Google-GeminiNotebook Google User-triggered fetch Generally not Google-Pinpoint Google User-triggered fetch Generally not Meta-ExternalFetcher Meta User-triggered fetch Not always MistralAI-User Mistral User-triggered fetch Yes Perplexity-User Perplexity User-triggered fetch Generally not Several jobs (6) Amazonbot Amazon Product improvement, may train Amazon AI models Yes Applebot Apple Search (crawled data may also help train Apple models) Yes CCBot Common Crawl Open web crawl archive Yes Google-CloudVertexBot Google Crawl requested by site owners for Vertex AI Agents Yes GoogleOther Google Generic crawl for various Google product teams Yes Meta-ExternalAgent Meta AI model training and content indexing Yes Control token (2) Applebot-Extended Apple Control token (no requests of its own) Read from robots.txt only Google-Extended Google Control token (no requests of its own) Read from robots.txt only Amazonbot, Amzn-SearchBot and Amzn-User: three Amazon agents, three rules Amazon's page now describes three agents with independent robots.txt settings. What each does, which may train Amazon AI models and which may ignore robots.txt. Page updated 27 October 2026 Amzn-SearchBot: what it does and what blocking it changes Amzn-SearchBot (Amazon): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 27 October 2026 Amzn-User: what it does and what blocking it changes Amzn-User (Amazon): user-triggered fetch. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 27 October 2026 Applebot-Extended: opting out of Apple's model training without leaving search Applebot-Extended is Apple's opt-out token for generative model training. What it does, why Applebot can still crawl you and what stays in Spotlight and Siri. Page updated 22 October 2026 Applebot: what it does and what blocking it changes Applebot (Apple): several jobs. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 Baiduspider: what it does and what blocking it changes Baiduspider (Baidu): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 Bingbot: Microsoft's crawler, five user agents, how to verify Bingbot is Bing's standard crawler. The five Microsoft crawler names, how to verify a request and where crawl rate is set, read from Bing's own help page. Page updated 30 October 2026 CCBot: what it does and what blocking it changes CCBot (Common Crawl): several jobs. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 ChatGPT-User: the fetcher that opens your page when someone asks ChatGPT-User visits a page when a person asks ChatGPT or a custom GPT about it. Why robots.txt may not apply, its user-agent string and 234 published IP ranges. Page updated 26 October 2026 Claude-SearchBot: Anthropic's search crawler and what blocking changes Claude-SearchBot indexes content to improve Claude's search results. Its robots.txt token, what a block does and how it differs from the other Claude robots. Page updated 26 October 2026 Claude-User: the agent that fetches pages when someone asks Claude Claude-User retrieves a page when a person asks Claude something. What Anthropic says disabling it does, and how it differs from ClaudeBot and Claude-SearchBot. Page updated 24 October 2026 ClaudeBot: Anthropic's training crawler, Crawl-delay and blocking ClaudeBot collects web content that could contribute to Anthropic's model training. The robots.txt group, Crawl-delay support and why IP blocking does not work. Page updated 23 October 2026 Diffbot: what it does and what blocking it changes Diffbot (Diffbot): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 DuckAssistBot: DuckDuckGo's real-time answer crawler and its 486 addresses DuckAssistBot fetches pages for DuckDuckGo's AI-assisted answers, which cite sources. Opt-out takes 72 hours; rankings are unaffected; 486 IP addresses listed. Page updated 25 October 2026 DuckDuckBot: what it does and what blocking it changes DuckDuckBot (DuckDuckGo): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 ExaSearchBot: what it does and what blocking it changes ExaSearchBot (Exa): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 ExaUser: what it does and what blocking it changes ExaUser (Exa): user-triggered fetch. What robots.txt does not control, how to refuse it and how to verify a request. Page updated 25 October 2026 Google-Agent: what it does and what blocking it changes Google-Agent (Google): user-triggered fetch. What robots.txt does not control, how to refuse it and how to verify a request. Page updated 25 October 2026 Google-CloudVertexBot: what it does and what blocking it changes Google-CloudVertexBot (Google): several jobs. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 Google-Extended: the robots.txt token for Gemini, not a crawler Google-Extended is a control token, not a crawler: no user agent, never in server logs. What it controls, what it leaves alone and why IP lists cannot verify it Page updated 6 November 2026 Google-GeminiNotebook: what it does and what blocking it changes Google-GeminiNotebook (Google): user-triggered fetch. What robots.txt does not control, how to refuse it and how to verify a request. Page updated 25 October 2026 Google-Pinpoint: what it does and what blocking it changes Google-Pinpoint (Google): user-triggered fetch. What robots.txt does not control, how to refuse it and how to verify a request. Page updated 25 October 2026 Googlebot: what it covers, its IP ranges and what a block removes Googlebot's robots.txt token controls Google Search and the products built on it. Its user-agent, the 317 IP ranges Google publishes and how to verify one. Page updated 25 October 2026 GoogleOther: what it does and what blocking it changes GoogleOther (Google): several jobs. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 GPTBot: what it does and what blocking it changes GPTBot crawls content that may be used to train OpenAI models. What a Disallow does, what it leaves alone, its user-agent and the 18 IP ranges OpenAI publishes. Page updated 6 November 2026 ICC-Crawler: what it does and what blocking it changes ICC-Crawler (NICT (Universal Communication Research Institute)): training. Robots.txt token, what a Disallow changes and how to verify a request. Page updated 25 October 2026 ImagesiftBot: what it does and what blocking it changes ImagesiftBot (Hive (ImageSift)): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 Kagibot: what it does and what blocking it changes Kagibot (Kagi): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 LinkupBot: what it does and what blocking it changes LinkupBot (Linkup): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 Meta-ExternalAgent: Meta's AI crawler and its four siblings Meta-ExternalAgent crawls for training foundation AI models or indexing. Meta's page, updated 21 May 2026, lists five agents and says two may bypass robots.txt. Page updated 25 October 2026 Meta-ExternalFetcher: what it does and what blocking it changes Meta-ExternalFetcher (Meta): user-triggered fetch. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 Meta-WebIndexer: what it does and what blocking it changes Meta-WebIndexer (Meta): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 24 October 2026 MistralAI-User, MistralAI-Index and MistralAI-Training explained Mistral documents three robots: a user-action fetcher, a search index agent and a training crawler. What each does, its robots.txt token and published IP files. Page updated 25 October 2026 MojeekBot: what it does and what blocking it changes MojeekBot (Mojeek): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 OAI-SearchBot: what it is and how to allow or block it OAI-SearchBot decides whether your pages can appear in ChatGPT search answers. Its robots.txt token, user-agent, 39 published IP ranges and what a block costs. Page updated 22 October 2026 Perplexity-User: what it does and what blocking it changes Perplexity-User (Perplexity): user-triggered fetch. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 PerplexityBot: how it differs from Perplexity-User and how to allow it PerplexityBot surfaces and links sites in Perplexity search; it is not used for foundation model crawling. User-agent, the 8 IP ranges published, WAF advice. Page updated 30 October 2026 Storebot-Google: what it does and what blocking it changes Storebot-Google (Google): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 YandexBot: what it does and what blocking it changes YandexBot (Yandex): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 YouBot: what it does and what blocking it changes YouBot (You.com): search. Robots.txt token, what a Disallow changes and how to verify a request, read from the operator's own page. Page updated 25 October 2026 ## Allow or block AI crawlers on 13 site platforms. CitedBy Source: https://citedby.eu/platforms How to allow or block AI crawlers on 13 site platforms One page per platform: where its own help page says the setting lives, what it does and does not cover, and the one thing that page says that the others do not, each read on the date shown. By CitedBy · Page updated 27 October 2026. method Each page names the platform's own help page, where the AI-crawler setting sits, what it covers and what it does not: Adobe Commerce, Cloudflare, Framer, Ghost, Netlify, PrestaShop, Shopify, Squarespace, Vercel, Webflow, Wix, WordPress.com, WP Engine. Adobe Commerce: robots.txt is in Content, Design, Configuration Adobe Commerce edits robots.txt in the Admin under Search Engine Robots. Cloud projects add pub/media steps, and Adobe's pages name no AI crawler. Page updated 27 October 2026 Cloudflare: AI bot policies for Search, Agent and Training Cloudflare's Block AI Bots page splits AI bots into Search, Agent and Training with Block, Block on pages with ads and Allow. New defaults from 15 Sept 2026. Page updated 27 October 2026 Framer: upload your own robots.txt with Static Files Framer generates a robots.txt for every published site. To change it, upload your own through Static Files on Pro or Enterprise, which replaces it. Page updated 27 October 2026 Ghost: add robots.txt to your theme to allow or block AI crawlers Ghost has no robots.txt setting in the admin. Its help says to save a robots.txt in your theme's root, which replaces the default. Plan needed on Ghost(Pro). Page updated 27 October 2026 Netlify: block AI crawlers with robots.txt and User Agent Blocker Netlify has no robots.txt setting: serve a file from your site root, or install the User Agent Blocker extension. Netlify's guide says to use both. Page updated 27 October 2026 PrestaShop: robots.txt generator writes one rule set for all bots PrestaShop's guide describes a Generate robots.txt button that replaces your file and uses User-agent: * only. Add AI crawler rules after generating. Page updated 27 October 2026 Shopify: allow or block AI crawlers, and what robots.txt misses Edit robots.txt.liquid to allow or block AI crawlers on Shopify. Shopify says a block affects open-web discovery only, not product data sent to ChatGPT. Page updated 27 October 2026 Squarespace: the AI crawler box blocks 26 bots but not ChatGPT Squarespace's Crawlers panel has one box blocking known AI crawlers. The 26 names it writes to robots.txt leave out ChatGPT's search and user bots. Page updated 27 October 2026 Vercel: the AI Bots ruleset in the Firewall (Log or Deny) Vercel blocks AI bots with a Firewall ruleset, not a robots.txt line. Where the AI Bots Ruleset is, the Log and Deny actions, and its self-updating bot list. Page updated 27 October 2026 Webflow: robots.txt rules and the Content-Signal header for AI Where Webflow's robots.txt editor is, which plan it needs and how its Content-Signal header separates AI training, search and AI input. Page updated 27 October 2026 Wix: block AI crawlers in the Robots.txt Editor (four names) Wix's help page tells you to paste four user agents into the Robots.txt Editor. Where the editor is, what the example leaves out and the nosnippet option. Page updated 27 October 2026 WordPress.com: AI opt-out is a privacy setting, not robots.txt WordPress.com treats AI opt-out as 'Prevent third-party sharing' under privacy settings. What its site-visibility guide says, reviewed 5 October 2026. Page updated 27 October 2026 WP Engine: robots.txt, 10-second default crawl-delay, Web Rules WP Engine's help page says a physical robots.txt overrides plugin files and sets a 10-second default crawl-delay. Web Rules Engine denies bots by user agent. Page updated 27 October 2026 ## AI visibility guides for online shops. CitedBy Source: https://citedby.eu/guides AI visibility guides for online shops How AI assistants find, read and name online shops, and what to fix. Each guide ends with an audit you can run on your own site, from €9. These guides cover where to edit robots.txt and find the sitemap on Shopify and WooCommerce, how to check product JSON-LD, ten technical checks you can run by hand, how to see whether ChatGPT, Perplexity and Gemini name your shop, and how the AI visibility tools compare on price. Best GEO and AI visibility audit tools for small businesses in 2026: which fits, and what each costs AI visibility and GEO tools compared on 28 Oct 2026: price, prompts and engines for Peec, Otterly, Profound, Ahrefs, Semrush, HubSpot and CitedBy. Page updated 2026-11-10 Why is my site not indexed by Google? What each reason in Google Search Console's Page indexing report means, in Google's own words, which ones you can fix, and what to do after the fix. Page updated 2026-11-06 How to set up Bing Webmaster Tools for a new site Add a new site to Bing Webmaster Tools: import it from Search Console or verify it by file, meta tag or DNS, submit the sitemap and check your key pages. Page updated 2026-11-04 How to check whether AI assistants recommend your shop and which of your pages they cite See by hand in 10 minutes whether ChatGPT, Perplexity and Gemini name your shop and which of your pages they link, and what to fix when they do not. Page updated 2026-11-03 How to see which of your pages Copilot cites: download your Bing AI Performance report Where to click in Bing Webmaster Tools to export which of your pages Copilot cites, and how CitedBy turns the file into a fix list ranked by citations. Page updated 2026-10-27 How to get your website cited by ChatGPT, Perplexity and Claude What a site owner controls to get cited by ChatGPT, Perplexity and Claude: crawler access, text in the HTML, one page per question. And what nobody controls. Page updated 2026-10-27 Technical SEO checklist for AI visibility: 10 checks for your online shop Ten technical checks that decide whether AI assistants and search engines can read your shop, each with a way to test it by hand in your browser. Page updated 2026-10-26 AI visibility checklist for Shopify, WooCommerce and more Where to find robots.txt, the sitemap and product data on Shopify, WordPress with WooCommerce and other shop platforms, and what to check for AI crawlers. Page updated 2026-10-21 ## AI visibility blog for online shops. CitedBy Source: https://citedby.eu/blog AI visibility blog for online shops Notes and data on how AI assistants treat online shops, plus our step-by-step guides. Each post ends with an audit you can run on your own site. The posts cover whether ChatGPT can read your website, whether llms.txt works, how to track AI referral traffic in GA4, how to check what AI says about your brand and what to do when an assistant shows a wrong price. What does ChatGPT say about your brand? 20 prompts to check it by hand A free 20-prompt test of what AI assistants say about your shop: do they name you, link you, name rivals instead, get your facts right? Includes a log sheet. Page updated 2026-11-11 Best GEO and AI visibility audit tools for small businesses in 2026: which fits, and what each costs AI visibility and GEO tools compared on 28 Oct 2026: price, prompts and engines for Peec, Otterly, Profound, Ahrefs, Semrush, HubSpot and CitedBy. Page updated 2026-11-10 Why is my site not indexed by Google? What each reason in Google Search Console's Page indexing report means, in Google's own words, which ones you can fix, and what to do after the fix. Page updated 2026-11-06 How to set up Bing Webmaster Tools for a new site Add a new site to Bing Webmaster Tools: import it from Search Console or verify it by file, meta tag or DNS, submit the sitemap and check your key pages. Page updated 2026-11-04 ChatGPT shows my old price. How to find where it read it and fix it ChatGPT quotes a price you changed months ago. The three places it can come from (your page, your feed, other sites), how to trace it and how to re-test. Page updated 2026-11-04 Does llms.txt work? What two studies and Google's own documentation say llms.txt is a proposed file for AI assistants. What the Ahrefs and SE Ranking studies found, what Google says, and when it is still worth adding. Page updated 2026-11-04 Is my website cited by ChatGPT? How to check it, step by step Four checks tell you if ChatGPT can reach, index, name and send visitors to your site, with numbers from our crawl of 370 launched sites. Page updated 2026-11-04 How a template flood cost a young site of ours 433 of 464 Google pages Search Console data from a young site of ours: 6 to 1,806 sitemap URLs in five days, then 612 to 44 impressions a day. The three checks we wrote. Page updated 2026-11-04 How to check whether AI assistants recommend your shop and which of your pages they cite See by hand in 10 minutes whether ChatGPT, Perplexity and Gemini name your shop and which of your pages they link, and what to fix when they do not. Page updated 2026-11-03 How to see which of your pages Copilot cites: download your Bing AI Performance report Where to click in Bing Webmaster Tools to export which of your pages Copilot cites, and how CitedBy turns the file into a fix list ranked by citations. Page updated 2026-10-27 How to get your website cited by ChatGPT, Perplexity and Claude What a site owner controls to get cited by ChatGPT, Perplexity and Claude: crawler access, text in the HTML, one page per question. And what nobody controls. Page updated 2026-10-27 Technical SEO checklist for AI visibility: 10 checks for your online shop Ten technical checks that decide whether AI assistants and search engines can read your shop, each with a way to test it by hand in your browser. Page updated 2026-10-26 My Lovable, Bolt or React site looks empty to AI. How to check it and fix it Many AI crawlers do not run JavaScript, so a client-rendered site can look empty to them. A 30-second test, what Lovable says, and the fixes by setup. Page updated 2026-10-26 How to fail a deploy when AI crawlers can't read your site A free command and GitHub Action that fail your build when robots.txt shuts out AI crawlers or the home page has almost no text. Setup, real output, limits. Page updated 2026-10-25 How to create an llms.txt file, step by step Write an llms.txt in six steps with a copy-paste template that follows the llmstxt.org format, publish it at your site root and check it with a free tool. Page updated 2026-10-24 Can ChatGPT read my website? Three things decide whether ChatGPT and other AI assistants can read your shop: robots.txt, the text in your HTML, and structured data. How to check each. Page updated 2026-10-23 Is GPTBot blocked on my site? Four places to check, with a test for each GPTBot can be stopped by robots.txt, your CDN, a firewall rule or your server. How to find out which, in minutes, and what blocking it does and does not change. Page updated 2026-10-23 GEO audit vs AEO audit vs SEO audit: what is different and which one you need Three names for overlapping checks. What each audit looks at, what Google says about AI features, and a short order of work for a small online shop. Page updated 2026-10-22 Cloudflare is blocking AI crawlers on my site. How to allow GPTBot and ClaudeBot Where Cloudflare's AI Crawl Control setting lives, which OpenAI and Anthropic crawler to allow for which goal, and how to confirm the change worked. Page updated 2026-10-21 AI visibility checklist for Shopify, WooCommerce and more Where to find robots.txt, the sitemap and product data on Shopify, WordPress with WooCommerce and other shop platforms, and what to check for AI crawlers. Page updated 2026-10-21 Is AI traffic converting? How to set up GA4 to see visits from ChatGPT and other assistants Build a custom AI assistants channel in GA4, check which sources really appear in your report, and compare conversions with other channels. Regex file included. Page updated 2026-10-21 ## Best GEO / AI visibility tools for small businesses, 2026 Source: https://citedby.eu/guides/ai-visibility-tools-compared Guides / Guide Best GEO and AI visibility audit tools for small businesses in 2026: which fits, and what each costs By CitedBy · Page updated 10 November 2026. 12 sources · method Explanation written with AI from the official source cited on this page. Short answer Pick by the job. To count how often assistants name your brand, use a tracker: Otterly.ai from €29 a month, Ahrefs Brand Radar from €47 or Peec AI from €85. To find what stops crawlers and search engines reading and indexing your pages, use an audit: CitedBy gives the exact fix per issue, from €9 once. Read 28 October 2026. On this page Trackers: they count how often assistants name you Audits and official reports: they show what blocks you What CitedBy has that the others do not Does llms.txt help? Which to pick How we compared Questions people ask Sources Trackers: they count how often assistants name you A tracker runs your list of prompts in the assistants every day and charts whether your brand appears, where, and who appears instead. The unit you pay for is prompts. Tool Price, read 28 October 2026 Prompts and engines Fits Otterly.ai Lite €29 a month, Standard €189, Premium €489; free trial 15, 100 or 400 prompts; ChatGPT, Google AI Overviews, Perplexity and Copilot on every plan Small teams that want the cheapest daily tracking Peec AI Starter €85 a month, Pro €205, Advanced €425; Enterprise on request 50, 150 or 350 prompts; you choose 3 models; Advanced adds several countries and Looker Studio SEO teams that report AI share of voice every week Ahrefs Brand Radar Custom Prompts from €47 a month; AI Visibility Index €179 a month AI Overviews, AI Mode, Perplexity, Gemini, Copilot and ChatGPT; paid Ahrefs plans include 5 to 20 prompts Teams that already work in Ahrefs Semrush SEO + AI Search SEO €112.52 a month billed yearly (€134 billed monthly), Starter €158.40 (€190), Pro+ €238 (€286) Starter tracks 50 prompts a day for 1 domain, Pro+ 100 Teams that want SEO and AI tracking in one suite Profound 7-day trial with 50 prompts a day on 3 engines; Enterprise has no public price Enterprise picks from 9 engines Large brands with a sales-led process Rankscale Essentials $17 a month billed yearly, Pro $99 17+ engines; Pro tracks up to 4,800 answers Teams that need many engines on a small budget HubSpot AI Search Grader Free, once; HubSpot AEO $50 a month Asks for company, location, product and industry; reads GPT-5.4 mini, Perplexity and Gemini A first look at how assistants describe a brand Audits and official reports: they show what blocks you An audit reads your own site. It says whether crawlers can fetch a page, whether a search engine will index it, and what to change. The official reports are free and come from the engines themselves. Tool Price, read 28 October 2026 What it gives Fits CitedBy €9 once for 1 site up to 500 pages; Starter €19 a month, Pro €49, Agency €99; 7-day free trial on the plans A crawl up to your plan's page limit (500, 1,000, 5,000 or 6,500) and the exact fix for each issue, with the lines to change; the monthly plans add daily checks that crawlers can still get in Shop owners, freelancers and agencies who want the fix list Ahrefs Webmaster Tools Free on sites you verify Site Audit with 170+ technical and on-page checks, 5,000 crawl credits a month per project Anyone who wants a free technical audit of their own site Google Search Console Free AI Overviews and AI Mode traffic is counted inside the Web search type of the Performance report Clicks and impressions from Google Bing Webmaster Tools, AI Performance Free, public preview since February 2026 Which of your pages Copilot cites, with the grounding queries behind each citation The only first-party view of Copilot citations Ahrefs and Semrush also run site audits inside their paid suites, so a team that already pays for one has a technical audit. What CitedBy has that the others do not The exact fix, per page. Each issue in a CitedBy report names the page, quotes the lines that cause it and says what to change. The sample report is our crawl of our own site, citedby.eu, on 1 November 2026: 6,496 pages and 9 issues, each with its fix. Checks for page floods and pages that will not be indexed. On 21 and 22 October one of our young sites put about 1,400 near-identical question pages live: its sitemap went from 6 URLs on 17 October to 1,806 on 22 October, and its Google impressions then fell by more than 90%. Our grader had scored that site A 96. After we added three checks, the live grader reads it C- 72 on 28 October: template_flood (most of the sitemap is one template), sitemap_publish_spike (many sitemap URLs dated the same day) and index_likely_not (sitemap pages that are blocked, noindexed, duplicated or nearly empty). The same change moved hexaciv.com from A+ 100 to B 86. We wrote the checks from this case, so the numbers show what they measure on our own sites. Data on how the web treats AI crawlers. On 23 October 2026, 7.9% of 10,426 popular EU websites blocked at least one crawler that builds AI answer indexes (OAI-SearchBot, Claude-SearchBot or PerplexityBot), and 18.4% blocked at least one of GPTBot, ClaudeBot, Google-Extended and CCBot. The method, the file and the per-country counts are in the EU AI-Access Index. What the assistants say about this very question. In our ChatGPT audit of 27 and 28 October 2026 we asked 28 questions about AI visibility tools, written the way buyers write them, up to four times each in fresh logged-out sessions. ChatGPT named CitedBy in 1 of the 96 answers (1.0%, 95% interval 0.0% to 3.4%). In 9 saved answers to four English tool-choice questions, Otterly appeared in 7, Profound in 7, Peec AI in 6, Semrush in 5, Scrunch in 3 and Ahrefs in 2. That is the field this page compares. Does llms.txt help? Not on the evidence we have. Ahrefs found that 97% of valid llms.txt files got no requests in May 2026, SE Ranking found no clear link to AI citations across about 300,000 domains, and Google says you do not need AI text files to appear in AI Overviews or AI Mode. We measured how common the file is: on 24 and 25 October 2026, 11.7% of 10,223 popular EU websites published one ( EU llms.txt Index ). A CitedBy audit lists a missing llms.txt as a notice, its lowest severity. The full reading is in Does llms.txt work?. Which to pick Your situation Pick Why You want 50 to 400 prompts run every day and charted Otterly.ai, Peec AI or Ahrefs Brand Radar Running your prompts daily is what they do, and each prices by prompt You already pay for Ahrefs or Semrush The AI features in that plan Ahrefs paid plans include 5 to 20 tracked prompts; Semrush Starter tracks 50 a day A large brand across many markets, with SSO or an API Profound or Semrush Advanced Both sell by quote: Semrush Advanced tracks unlimited prompts, Profound Enterprise picks from 9 engines You want one free look at how assistants describe you HubSpot AI Search Grader Free, once, no account You want to know whether Copilot and Google show your pages Bing AI Performance and Search Console First-party numbers, free Crawlers get blocked, pages are thin, copied or not indexed, or you need a fix list CitedBy, or Ahrefs Webmaster Tools for a free audit of your own site An audit names the page and the change; a tracker reports the symptom You run an agency with client sites CitedBy Agency (€99 a month, 25 sites, white-label PDF) or an Otterly or Peec agency plan Pick by whether clients want fixes or a prompt dashboard If you will use both kinds, run the audit first: an assistant cannot quote a page that its crawler could not fetch. How we compared We read each vendor's own pricing and product page on 28 October 2026 and list what it showed; the page of each tool is under Sources. Prices are list prices, so check the vendor's page before you buy. Otterly's page showed euro prices in our browser and dollar prices ($29, $189, $489) in a plain fetch. CitedBy's figures come from our plans page, and the figures about our own sites come from our own crawls and Search Console. How to check by hand shows the same test without any tool. Questions people ask What is a GEO audit tool? It reads your website and reports what stops AI assistants and search engines from reaching and reading it, with the change to make. A tracker does something else: it asks assistants questions and counts who they name. Do I need a tracker and an audit? Start with the audit if you have never checked crawler access or indexing. Add a tracker once there is something to measure and someone who will read it every week. Is there a free way to check? Yes. Search Console and Bing Webmaster Tools are free for your verified site, the HubSpot grader is free once, Ahrefs Webmaster Tools audits your own site free, and the CitedBy free check reads 10 pages with no account. Does the cheapest tracker give the same data as the expensive one? Prompt limits and engines differ. Otterly's three plans all cover the same four engines and differ in prompts, 15 to 400; Peec's plans differ in prompts, 50 to 350, and Advanced adds several countries. Sources Each vendor's page, read on 28 October 2026. Otterly.ai pricing Peec AI pricing Profound pricing Ahrefs Brand Radar Ahrefs Webmaster Tools Semrush pricing Rankscale pricing HubSpot AI Search Grader Google Search Central: AI features and your website Bing Webmaster Blog: AI Performance in Bing Webmaster Tools Ahrefs llms.txt study Search Engine Journal: SE Ranking study of about 300,000 domains, 20 November 2025 CitedBy plans Keep reading Profound vs Peec vs Otterly How to check whether AI recommends your shop Does llms.txt work? See which pages Copilot cites (Bing AI Performance) EU AI-Access Index ## See which pages Copilot cites: Bing AI Performance report Source: https://citedby.eu/guides/bing-ai-performance-export Guides / Bing and Copilot How to see which of your pages Copilot cites: download your Bing AI Performance report By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer In Bing Webmaster Tools, pick your verified site, open AI Performance and choose a date range of up to six months. Click Download at the top for the daily file, then Download all on the Pages and Grounding Queries tabs. Upload the files in CitedBy under Settings, Copilot citations. Microsoft offers no API for it. On this page What does the Bing AI Performance report show? How do you download it? What does CitedBy do with the file? Questions people ask Sources What does the Bing AI Performance report show? It shows how often pages of your site were cited in Microsoft Copilot, Bing's AI summaries and some Microsoft partner experiences. You get total citations, the pages that were cited, a timeline and the grounding queries: the short phrases the AI searched before it cited you. Newer versions add each query's intent, topic and citation share, and a Topics Portfolio with opportunity queries, where other sites are cited and yours is not ( Bing Webmaster Blog ). Microsoft samples this data. A site with little activity may show little or nothing. It covers Microsoft's assistants only, not ChatGPT, Gemini or Perplexity, and a citation is not a visit. How do you download it? Verify your site in Bing Webmaster Tools. The report only exists for verified sites. Open AI Performance for your site. It sits in the left menu. Choose the date range. Ranges run from 7 days to 6 months, or a custom range. Click Download at the top for the daily file. Open the Pages tab and click Download all. Do the same on Grounding Queries. Each button saves the list that is open, so you end with three files. Keep them as they come. Upload the files in CitedBy. Open your site's Settings, then Copilot citations, and drop the files in. Several files at once are fine. The Pages file is the one that ranks your fixes; the daily file adds the period and the queries file adds what Copilot searched for. If a Topics Portfolio shows for your site, export that view as well: its opportunity queries become pages to write or fix. What does CitedBy do with the file? CitedBy joins your cited pages to the latest audit of the same site, by address. It lists the audit's fixes on pages Copilot already cites first, ranked by citations. For each opportunity query it names the page that should answer it, or says that no crawled page does. Upload a later export and it shows what changed between the two. We keep counts, page addresses and query words, not the file, and you can remove them in the same place. Without an upload nothing is estimated: the card reads "Not measured". Questions people ask Is there an API for the AI Performance report? No. Microsoft's Webmaster API has no method for it, which is why the download is manual ( Microsoft Q&A ). Does the import need a paid CitedBy plan? Yes. It is part of Starter and up. The Bing report itself is free. Sources Introducing AI Performance in Bing Webmaster Tools AI Performance help Find what stops AI assistants reading your shop The audit crawls your site up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. Audit my site Audit from €9, paid once. 14-day money-back guarantee. Keep reading Technical SEO checklist for AI visibility Can ChatGPT read my website? What changed in search and AI engines ## How to set up Bing Webmaster Tools for a new site Source: https://citedby.eu/guides/bing-webmaster-tools Guides / Bing and Copilot How to set up Bing Webmaster Tools for a new site By CitedBy · Page updated 4 November 2026. 12 sources · method Explanation written with AI from the official source cited on this page. Short answer Go to bing.com/webmasters and sign in with a Microsoft, Google or Facebook account. If the site is already verified in Google Search Console, import it from there. Otherwise type its address, then verify it with the BingSiteAuth.xml file, a meta tag or a DNS record. Submit your sitemap under Sitemaps and inspect your key pages. On this page The six steps What do you need before you start? How do you add the site? How do you verify a site by hand? How do you submit the sitemap? How do you check whether a page is indexed? Why is a new page not in Bing yet? How do you tell Bing about new and changed pages? What is worth opening once the site is in? Questions people ask Sources The six steps Sign in at bing.com/webmasters with a Microsoft, Google or Facebook account. Add the site: import it from Google Search Console, or type its address and click Add. Verify that you own it with the BingSiteAuth.xml file, a meta tag or a DNS record. An imported site is already verified. Submit your sitemap under Sitemaps, then name it in your robots.txt. Inspect your key pages in URL Inspection: the home page and the pages you most want found. Send new and changed pages through IndexNow from then on. You are done when the site is verified, your sitemap's status reads Success on the Sitemaps page, and URL Inspection answers for your home page. Bing says reports usually show data about 48 hours after a site is added ( Bing help: add and verify a site ). The sections below take each step in turn. What do you need before you start? Three things: an account to sign in with, a way to change your site (upload a file, edit the , or edit DNS records at your domain provider), and the address of your sitemap, often /sitemap.xml. Open it in your browser first. If it does not load, fix that before you add the site: the sitemap is how Bing learns about pages that nothing links to yet. How do you add the site? You have two ways in ( Bing help: add and verify a site ): Import from Google Search Console. Bing asks for permission to read your Search Console account, lists the verified sites in it with their sitemaps, and you pick the ones to bring over, up to 100 at a time. Ownership and sitemaps come with them. Bing checks your Search Console account again from time to time and imports any new sitemap it finds there. Add your site manually. Type the address in the box and click Add. You can add a whole domain ( example.com ) or one section of it ( example.com/blog/ ). Then verify it. If you already have sites in the account, add a new one from the site menu on the left. How do you verify a site by hand? Bing shows the methods your domain supports. Pick one and follow the screen ( Bing help: add and verify a site ): XML file. Download BingSiteAuth.xml and upload it to the root of the site, so it opens at yourdomain.com/BingSiteAuth.xml. In a Next.js or Vite project, put it in the public folder, which is served from the root ( Next.js docs, Vite docs ). Meta tag. Copy the tag Bing shows and paste it at the end of the of your home page, then publish. CNAME record. At your domain provider, add the CNAME record with the code Bing gives you. DNS auto verification. Shown only if your DNS provider supports Domain Connect: you log in to the provider when asked and Bing adds the record for you. Pick the first method that fits your site. Then finish the on-screen step to verify. If verification fails, check the method you chose: the file opens in a private browser window at exactly /BingSiteAuth.xml on the address you added (with or without www, as you typed it); the meta tag is in the HTML your server sends: open the page, choose View source, and search for msvalidate.01. If it is missing there, switch to the file or the DNS record; the DNS record is saved at the provider that actually serves your domain. If Bing does not see it yet, wait and try verifying again. Leave the file, tag or record in place after verifying. How do you submit the sitemap? Open Sitemaps in the left menu and click Submit sitemap at the top right. Paste the full address, for example https://yourdomain.com/sitemap.xml, and click Submit. The dialog lists the formats it accepts: XML, RSS 2.0, mRSS, Atom 0.3 and 1.0, and text files with one URL per line. The Submit sitemap dialog, captured 4 Nov 2026. The table then shows each sitemap's last submit and crawl dates, its status and how many URLs Bing found in it. Click a sitemap to see the exact error or warning to fix ( Bing help: Sitemaps ). Our own site, citedby.eu, on the Sitemaps page, captured 3 Nov 2026. Also add one line to your robots.txt so Bing finds the sitemap by itself: Sitemap: https://example.com/sitemap.xml List only addresses you want in search: pages that answer 200, carry no noindex and name themselves as canonical. How do you check whether a page is indexed? Open URL Inspection, paste the address and click Inspect. It shows whether the page is in Bing's index and when Bing discovered, crawled and indexed it, and its Live Check tests whether Bingbot can crawl the page now ( Bing Webmaster Blog ). URL Inspection takes addresses of the selected site only. From outside the tool, a search for url: followed by the address shows whether Bing returns it ( Bing help: why is my site not in the index? ). Submit URLs sends addresses one per line. The quota is 10,000 URLs per domain per day and resets at midnight UTC. Submitting the same address again does not speed it up ( Bing help: URL submission ). Why is a new page not in Bing yet? Bing's own reasons, split by who can check them, plus two checks of ours: a redirect or error, and a canonical naming another address. Bing's help lists these reasons ( Bing help: why is my site not in the index? ): The site is new. Bing still has to find links to it and crawl it, even after you submit a sitemap or URLs. No other site links to it. Bing needs at least one link pointing at your site to find it, and prefers a few from quality sites. Bingbot cannot crawl it reliably. URL Inspection shows whether Bingbot can fetch your home page and other pages. robots.txt stops Bingbot. The page carries a noindex robots meta tag. You blocked the address with the Block URLs tool in your own account. The content falls under Bing's quality bar, for example copied text, a page that quickly redirects visitors, or very thin pages. The site was penalised or carries malware. Bing alerts owners in Webmaster Tools when it detects malware. Two of them you can check from outside before Bing crawls anything: the robots.txt rule and the noindex. Check two more while you are there, though they are not on Bing's list: a page that redirects or answers with an error, and a canonical tag that asks Bing to index another address instead. Our free Bing index check reads them for up to 10 of your URLs, plus your sitemap and whether a verification file or tag is on the site. Whether a page is actually in Bing's index only Bing can show: URL Inspection, or the url: search above. How do you tell Bing about new and changed pages? Use IndexNow. Bing's help calls it the strongly recommended way to submit URLs and says manual submission and its API may be deprecated later ( Bing help: URL submission ). By hand, it works like this ( IndexNow documentation ): Make your own key: 8 to 128 characters, only letters, numbers and dashes. Do not copy the example below; anyone who knows your key can submit URLs for your site. Publish a text file named after the key at the root of your site, containing only the key, for example https://example.com/YOURKEY.txt. Open it in a browser: it must show the key and nothing else. When a page is added, changed or deleted, request this address once, with the page address URL-encoded: https://api.indexnow.org/indexnow?url=https%3A%2F%2Fexample.com%2Fpage&key=YOURKEY The answer code tells you what happened ( IndexNow documentation ): Code Meaning 200 URL received 202 URL received, the key is still being checked 403 key not valid: the file is missing or does not hold the key 422 the URL is not on your host, or the key does not fit the format 429 too many requests One request reaches every engine in the protocol; the FAQ lists Amazon, Bing, Naver, Seznam.cz, Yandex and Yep, and Google is not among them ( IndexNow FAQ ). Many CMS plugins and CDNs send IndexNow for you, so check yours before you build it. The IndexNow tab in Webmaster Tools shows what arrived. Send only addresses whose content changed. A ping tells the engine that something changed; it still decides what to crawl and index. What is worth opening once the site is in? The left menu, captured 3 Nov 2026. Search Performance: the queries and pages that bring you Bing impressions and clicks, with up to 16 months of history ( Bing Webmaster Blog ). AI Performance: how often Microsoft Copilot and Bing's AI summaries cite your pages, with a sample of the grounding queries behind those citations. It has been in public preview since February 2026 and is marked Beta in the menu ( Bing Webmaster Blog ). Our guide shows how to download that report. robots.txt tester: test an address against your robots.txt as Bingbot reads it ( Bing Webmaster Blog ). Site Scan: an on-demand crawl of your own site that sorts technical SEO issues into errors, warnings and notices ( Bing help: Site Scan ). Questions people ask Is Bing Webmaster Tools free? Yes. You sign in with a Microsoft, Google or Facebook account and add sites you can verify, up to 1,000 per account. Do I need it if I already use Google Search Console? Yes, if you want Bing's own data: how Bing crawls and indexes your site and what it sends you is reported in Bing Webmaster Tools. Microsoft also documents that Microsoft Copilot and Copilot Chat send the web searches they make to Bing ( Microsoft Learn ). Which verification method should I pick? Import from Search Console if the site is there. Otherwise the XML file when you can upload files, and the DNS record when you cannot. Use the meta tag only when you control the HTML your server sends. How long until Bing shows data? Bing says reports usually fill in about 48 hours after you add a site. Indexing a new site can take longer: Bing has to find links to it and crawl it first. Sources Bing help: add and verify a site Bing help: Sitemaps Bing help: URL submission Bing help: why is my site not in the index? Bing help: Site Scan Start using Bing Webmaster Tools to improve your site visibility (sign-in accounts) Introducing the URL Inspection tool The robots.txt tester Search Performance extended to 16 months Introducing AI Performance in Bing Webmaster Tools IndexNow documentation and FAQ Microsoft Learn: Copilot and web search Check what Bing will find before it crawls Type your domain. We read your robots.txt and sitemap and check up to 10 of your URLs for what stops Bingbot: a block, a noindex, a redirect or a canonical that points elsewhere. Check my site for Bing Result in seconds. No account, no card. Keep reading Can Bing index my pages? Free check See which pages Copilot cites (Bing AI Performance) Is my site findable by Google, Bing and ChatGPT? Can ChatGPT read my website? ## Do AI assistants cite your shop? Which pages, and how to check Source: https://citedby.eu/guides/check-if-ai-recommends-your-shop Guides / Guide How to check whether AI assistants recommend your shop and which of your pages they cite By CitedBy · Page updated 3 November 2026. method Explanation written with AI from the official source cited on this page. Short answer Ask each assistant the questions your buyers ask, twice, with web search on. Count how often your shop is named and which of your pages it links. CitedBy is a site audit, run once or on a schedule: it checks whether AI crawlers can reach and read your pages, from €9, with the exact fix for each issue. On this page Do it by hand in 10 minutes Keep a log Why one answer can mislead Do it with the site audit Check you are reading the right company If you are not named Questions people ask Sources Do it by hand in 10 minutes Write 5 questions a buyer would type, for your product and country. Example: "best online shop for natural latex mattresses in Belgium". Do not put your shop name in them. Ask each question in ChatGPT, Perplexity and Gemini, logged out, with web search on. Ask each question a second time. Answers change between runs. For each answer note two things: is your shop named in the text, and is your domain in the links. When it is, write down which of your pages the link points to. Add it up: named in 3 of 40 answers is a different situation from 0 of 40. Keep a log One row per answer, so you can count at the end: Question Assistant Run Shop named in the text? Domain linked? Page linked best natural latex mattress, Belgium ChatGPT 1 yes no none best natural latex mattress, Belgium ChatGPT 2 no no none Separate "named" from "linked". An assistant can describe your shop without linking you, and it can link your page without using your name. Both count, but they are different problems: the first is about whether it knows you, the second about which of your pages it trusts. Why one answer can mislead The same question gives different shops on different runs. In our own runs of 13 and 14 October 2026, seven category-and-country cells with about 16 answers each, two fresh runs of the same question named the same shops only 40 to 79% of the time, depending on the cell. If your first answer names you, ask again before you believe it, and if it does not, ask again before you give up. Do it with the site audit The CitedBy site audit crawls your site up to your plan's page limit and tells you whether AI crawlers can read it and whether your site answers the questions buyers ask, with the exact fix for each issue. It does not ask assistants whether they name you; the manual method above does that. Check you are reading the right company Ask each assistant "what is [your shop name]" first. If it describes another business with a similar name, fix that before anything else: add an About page with your legal name, country and domain, put Organization data in JSON-LD on the home page, and link the same facts from your listings. If you are not named The fixes that tend to matter are on your own site (a page that answers the buyer question directly, product data with price and availability) and off it (the pages assistants cite for your category, such as comparison articles and directories). Our sample audit shows what that looks like for one shop. The About page explains how we count. The audit starts at €9 with a 14-day money-back guarantee. Questions people ask How many questions should I ask? Five questions asked twice in three assistants gives 30 answers, enough to see whether you are named never, sometimes or often. More questions give a steadier picture. Why ask each question twice? Answers change between runs, so one answer can mislead. Log both and count them separately. Does the audit tell me if assistants name my shop? No. It checks whether your pages can be reached and read. The manual method above shows whether you are named today. Sources OpenAI: crawlers and user agents CitedBy methodology Keep reading Check what AI says about your brand, 20 prompts AI visibility tools compared A sample report Profound vs Peec vs Otterly compared How to get cited by ChatGPT, Perplexity and Claude ## AI visibility checklist for Shopify, WooCommerce and more Source: https://citedby.eu/guides/cms Guides / Guide AI visibility checklist for Shopify, WooCommerce and more By CitedBy · Page updated 21 October 2026. 4 sources · method Explanation written with AI from the official source cited on this page. Short answer On any shop platform, check four things: robots.txt does not block AI crawlers, the sitemap lists your products, product pages carry price and availability as JSON-LD, and product text is in the raw HTML. Shopify and WordPress generate the first two for you. Our audit tests all four on every page up to your plan's page limit, from €9. On this page Which checks apply on every shop platform? How do I check AI crawler access on Shopify? Where is the sitemap on Shopify and WordPress? How do I check product data and text on my platform? Which platform setting is easiest to miss? What does the audit check? Questions people ask Sources Which checks apply on every shop platform? The same four questions apply whatever you sell on. Open yourshop.com/robots.txt and check that nothing blocks the crawlers you want. Open yourshop.com/sitemap.xml and check that products are listed. View the source of a product page and search for application/ld+json to see the product data. Then search the same source for a sentence from the product description. If it is missing, a script adds it after loading and many crawlers never see it. The platform decides where you change each of these. The sections below say what the official help pages confirm and leave the rest to your own settings. How do I check AI crawler access on Shopify? Shopify creates a default robots.txt for every store. Shopify's help page says the file sits at the root of your primary domain, and that you can customise it by adding a robots.txt.liquid template to your theme. It calls this an unsupported customisation that Shopify Support cannot help with, so ask a Shopify Partner or a developer if you are unsure ( Shopify: editing robots.txt.liquid ). The same page explains that robots.txt rules are advisory, and that blocking AI crawlers there affects only open-web discoverability. Product data sent through Shopify Catalog to agentic storefronts you have activated is not stopped by it. Where is the sitemap on Shopify and WordPress? Shopify generates sitemap files automatically at yourshop.com/sitemap.xml. They link to separate sitemaps for products, collections, blogs and pages and update when you add content. Search engines cannot read the sitemap if the store is in private mode ( Shopify: find your sitemap ). WordPress, which WooCommerce runs on, has exposed a sitemap index at /wp-sitemap.xml since version 5.5, and the robots.txt it serves references that index ( WordPress core announcement ). An SEO plugin may replace it with its own sitemap address, so open both and see which one your site answers with. How do I check product data and text on my platform? Shopify's guidance on product pages for AI systems asks for clear titles, price and availability shown prominently, detailed specifications, structured data and product attributes. It adds that detail meant only for AI systems can sit lower on the page ( Shopify: optimizing your store for AI ). There is no official help page I could confirm that states exactly which structured data a given theme or plugin outputs on WooCommerce, Wix, Squarespace, PrestaShop or Magento. So check your own product page source. If application/ld+json is missing or shows the wrong price, look in your platform's settings or your SEO plugin for a structured data option, and compare the values against the page. Text hidden behind tabs, accordions or "read more" buttons is fine if it is in the HTML. It is a problem if a theme or app loads it with JavaScript only after the page opens. Which platform setting is easiest to miss? Platform Where to look What to confirm Shopify robots.txt.liquid in the theme code Default file is intact or edits are deliberate Shopify Online store preferences Store is not in private mode WordPress with WooCommerce /wp-sitemap.xml or your SEO plugin One sitemap, listing products Any platform CDN or firewall rules Crawlers are not refused by name Any platform Product page source JSON-LD price matches the page What does the audit check? Platform dashboards show settings. A crawler sees the result. The CitedBy audit crawls your shop up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. It starts at €9, paid once, with a 14-day money-back guarantee. Questions people ask Can I edit robots.txt on every platform? No. On Shopify you can, through the robots.txt.liquid template. On other platforms it depends on the platform and plan, so look in your platform's settings or help pages. Does a sitemap plugin matter for AI assistants? A sitemap helps crawlers find your pages, but assistants also need to be allowed in and to read the page. See the technical SEO checklist. My theme loads descriptions with JavaScript. What now? Ask the theme or app developer for a version that outputs the text in the HTML, or move the key facts into the standard product description field. Sources Shopify: editing robots.txt.liquid Shopify: finding and submitting your sitemap Shopify: optimizing your store for AI WordPress core: XML sitemaps in 5.5 Keep reading Technical SEO checklist for AI visibility Is GPTBot blocked on your site? Can ChatGPT read my website? ## How to get cited by ChatGPT, Perplexity and Claude Source: https://citedby.eu/guides/get-cited-by-chatgpt Guides / Guide How to get your website cited by ChatGPT, Perplexity and Claude By CitedBy · Page updated 27 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer Nobody can make an assistant cite a page. A site owner controls three things: whether the assistant's search crawler can fetch the page, whether the answer is in the HTML the server sends, and whether one page answers one question directly. Those make a page eligible to be cited. Whether an assistant then names it is the assistant's choice. On this page What you control What nobody controls Blogs and local businesses Check where you stand Questions people ask Sources What you control Access for the search crawlers. Each assistant has a crawler for search and a separate one for training. ChatGPT search uses OAI-SearchBot, Perplexity uses PerplexityBot and Claude uses Claude-SearchBot. Blocking a training crawler such as GPTBot or ClaudeBot does not block these. Open your robots.txt, then ask your server: a firewall or CDN can answer 403 to a crawler that robots.txt allows. The free AI crawler checker tests both, and /bots has the robots.txt lines to copy. Text in the HTML. Many crawlers do not run scripts. A page whose text is added by JavaScript reaches them as an empty shell. Open "View page source" and search for the sentence that answers the question; if it is not there, a crawler that does not run scripts cannot quote it. One page per question. A page that states the answer in its first lines, with the facts a buyer asks about (price, availability, who it is for), is easier to quote than a page that touches ten topics. Write the page for one question, in the words people ask it. Being in a search index. ChatGPT search draws on search-engine results from partners, and Bing is the one most often named. Add your site to Bing Webmaster Tools and Google Search Console and submit your sitemap, so the pages can be found at all. What nobody controls Which pages an assistant names changes from one run of the same question to the next. OpenAI does not publish a citation report for site owners, so you see its crawlers' requests and visits from chatgpt.com in your server logs and the rest only by asking the questions yourself. A mention without a link may send nobody to your site. Files such as llms.txt are cheap to add, but no major assistant has confirmed that it reads them. Be wary of anyone who sells a guaranteed citation. The part that can be fixed is access, text and structure, and that is the part to do first. Blogs and local businesses A blog helps when a post is the page that answers a question directly; more posts on the same topic do not add reasons to quote any of them. A local business gets the same three controls and one more habit: put your name, address, opening hours and what you sell in the HTML of a page of your own, so that the facts an assistant gives about you come from you. Check where you stand Run the free AI crawler checker on your domain: it shows the robots.txt rule and the live response for each crawler. Ask the questions your buyers ask in each assistant, twice, and note which of your pages are linked: how to check by hand. For the whole site, CitedBy audits your pages up to your plan's page limit and gives the exact fix for each issue, from €9 paid once. Questions people ask How long until ChatGPT cites a new page? We cannot give a number. The page must first be crawled and indexed, and answers change from run to run. Does blocking GPTBot stop ChatGPT citing me? Not by itself. GPTBot is the training crawler; OAI-SearchBot is the one used for search, and ChatGPT-User fetches a page when a person asks about it. Check each rule on its own. Do I need an llms.txt file? No major assistant has confirmed that it reads one. It is cheap to add and cheap to check, but access and text matter more: see does llms.txt work?. Sources OpenAI: Overview of OpenAI Crawlers Perplexity: Perplexity Crawlers Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? Keep reading How to check whether AI recommends your shop Can ChatGPT read my website? robots.txt for AI crawlers Free AI crawler checker ## Site not indexed by Google? Each status explained Source: https://citedby.eu/guides/site-not-indexed Guides / Google Search Console Why is my site not indexed by Google? By CitedBy · Page updated 6 November 2026. 9 sources · method Explanation written with AI from the official source cited on this page. Short answer In Search Console, open Indexing > Pages and read the reason in the "Why pages aren't indexed" table. A noindex you meant, or a duplicate, needs nothing. A server error, redirect error or 404 on a page that should exist is yours to fix. "Crawled - currently not indexed" is, in Google's words, "a common message for many sites". On this page Start with one question: is the page really missing? Where to find the reason Find your status Reasons where your site told Google to skip the page Reasons where the page answered with an error Reasons where Google has not indexed the page yet Reasons about duplicates and canonicals Indexed, with a warning After the fix Questions people ask Sources Start with one question: is the page really missing? For a single page, use the URL Inspection tool in Search Console: paste the address and read the verdict. Google's help says the Page indexing report is not the place to look up one page ( Google: Page indexing report ). URL is on Google: the page is indexed. URL is not on Google: Google: it "means that the URL can't appear in Search results". Open the Page indexing section of the result: the reason is generally under Crawl ( Google: URL Inspection tool ). URL is unknown to Google: Google "hasn't seen this URL before". Google's fix: run the live test, fix what it finds, then request indexing ( Google: URL Inspection tool ). For a small site, Google says you may not need the report at all: "If your site has fewer than 500 pages, you probably don't need to use this report." Search Google for site:example.com to see a sample of the pages it knows, or for site: followed by the exact address of one page. Google says to open the report only if those searches return no results ( Google: Page indexing report ). If a brand-new site shows nothing at all, Google needs to find it first: "you must either submit a sitemap or page crawl request, or else Google must find a link to your page somewhere." After that, "it can take some time (up to a few weeks) before Google crawls some or all of your site." Google's advice for a new website is to request indexing of the home page first ( Google: Page indexing report ). Where to find the reason In Search Console, open Indexing > Pages. The top of the report shows how many pages are indexed and not indexed, and the date of its last update. Our own site, citedby.eu, captured 4 Nov 2026. The counts are from the report's last update, 23 Oct 2026. Below it, the table Why pages aren't indexed lists one row per reason, with a Source column. Google says: "In general, you can fix only issues where the source is listed as 'Website'." Click a row to see the addresses it covers. The list shows at most 1,000 examples per reason ( Google: Page indexing report ). The same report's reasons on a narrow screen. The Pages column sits off to the right; the table below gives its counts. The same six rows with their page counts, from the report's last update on 23 Oct 2026. The source is Website for the first five and Google systems for the last. Reason Pages Excluded by ‘noindex’ tag 159 Blocked by robots.txt 60 Alternate page with proper canonical tag 38 Page with redirect 10 Not found (404) 4 Crawled - currently not indexed 115 The report does not always use the names on Google's help page. On our own site it shows "Excluded by ‘noindex’ tag" where the help page says "URL marked 'noindex'", and "Blocked by robots.txt" where the help page says "URL blocked by robots.txt". The headings below use the report's name where we have seen it, and the help page's name otherwise. Read the row before you act. Google's help puts it plainly: "It's fine for a URL not to be indexed for the right reasons", for example a robots.txt rule you meant, a noindex tag, a duplicate, or a 404 for a page you removed on purpose ( Google: Page indexing report ). Find your status Status in the report First step Excluded by ‘noindex’ tag Nothing, if you meant it Blocked by robots.txt Nothing, if you meant it Page with redirect Nothing, if the move was planned Alternate page with proper canonical tag Nothing Blocked by page removal tool Check the Removals page Server error (5xx) A fix on your server Redirect error A fix to the redirect Not found (404) A fix if the page should exist Soft 404 A real 404, or more content Blocked due to unauthorized request (401) Remove the login, if public Blocked due to access forbidden (403) Let Googlebot through URL blocked due to other 4xx issue Debug in URL Inspection Crawled - currently not indexed Inspect your key pages Discovered - currently not indexed Check links and sitemap Duplicate without user-selected canonical A canonical tag, if you prefer one Duplicate, Google chose different canonical than user Compare the two canonicals Duplicate, submitted URL not selected as canonical Compare the two canonicals Indexed, though blocked by robots.txt Decide: block or index Page indexed without content Inspect the page Reasons where your site told Google to skip the page Google's help counts a robots.txt rule you meant, a noindex tag and a duplicate among the right reasons for a page to stay out ( Google: Page indexing report ). Check that each address really is one you want out of Google. Excluded by ‘noindex’ tag The help page calls this "URL marked 'noindex'". Google found a noindex rule in the page or its HTTP header and left the page out. If you want it indexed, remove the rule, publish, then run Test live URL in URL Inspection: under Indexing allowed? the noindex should be gone. Then click Request indexing ( Google: Page indexing report ). To find the rule yourself, Google says to search the page source or response headers for the word "noindex" ( Google: Page indexing report ). In a response header it appears as X-Robots-Tag ( Google: Block Search indexing with noindex ). Blocked by robots.txt The help page calls this "URL blocked by robots.txt". Your robots.txt stops Googlebot from fetching the page. Google says these URLs "are probably intentionally blocked" ( Google: Page indexing report ). If the block is a mistake, remove the rule. If you want the page kept out of Google, robots.txt is the wrong tool. Google: "it is not a mechanism for keeping a web page out of Google", and a blocked page "can still be indexed if linked to from other sites" ( Google: Introduction to robots.txt ). Use noindex instead, and leave the page crawlable: "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule" ( Google: Block Search indexing with noindex ). Page with redirect Google: "This is a non-canonical URL that redirects to another page. As such, this URL will not be indexed. The target URL of the redirect might or might not be indexed, depending on what Google thinks about that target URL." ( Google: Page indexing report ). For an old address you moved on purpose, that is the expected result. Check the target address on its own in URL Inspection. If the redirecting address is still in your sitemap or your own links, replace it with the final address. Alternate page with proper canonical tag Google uses this for an AMP page with a desktop canonical, or a mobile and desktop pair of the same page. Google's words: "This page correctly points to the canonical page, which is indexed, so there is nothing you need to do." Google adds that Search Console does not detect alternate language pages ( Google: Page indexing report ). Blocked by page removal tool Google names this status in its URL Inspection help and says: "Check this in the Temporary Removals page instead" ( Google: URL Inspection tool ). In the Search Console menu that page sits under Indexing > Removals, as the left menu of our own account showed on 4 Nov 2026. Reasons where the page answered with an error Google reached your server and got an error, or a page that looks like one. Fix these when the page should exist. For the 401 case Google says: "You can verify this error by visiting the page in incognito mode" ( Google: Page indexing report ). Server error (5xx) Your server answered with a 500-level error when Google asked for the page. Google's help adds that Googlebot may have hit a timeout or a busy site. Its steps: check the host status in the Crawl Stats report, try the address in URL Inspection's live test, and make sure the server is not down, overloaded or misconfigured. A live test can pass even though Google's crawl failed, because server errors can be transient ( Google: Page indexing report ). Redirect error Google lists four causes: a redirect chain that was too long, a redirect loop, a redirect address that grew past the maximum URL length, or a bad or empty address in the chain. Google suggests a web debugging tool such as Lighthouse for the details ( Google: Page indexing report ). Our advice: redirect each old address straight to its final page in one step. Not found (404) The page answered 404. That is fine for a page you removed with no replacement: "404 responses are not necessarily a problem". If the page moved, add a 301 redirect to its new address. Google may keep trying the old address for a while: "there is no way to tell Googlebot to permanently forget a URL" ( Google: Page indexing report ). Soft 404 Google: the page "returns a user-friendly 'not found' message but not a 404 HTTP response code". Google recommends a real 404 code for pages that do not exist, and more content on the page if it is a real page ( Google: Page indexing report ). In single-page apps that route in the browser, Google says meaningful status codes "can be impossible or impractical". For those, it gives two ways out: a JavaScript redirect to an address where the server answers 404 (for example /not-found ), or a added to error pages ( Google: JavaScript SEO basics ). Blocked due to unauthorized request (401) The page asked Googlebot to log in. Google's fix: "either remove authorization requirements for this page, or else allow Googlebot to access your pages by verifying its identity" ( Google: Page indexing report ). Blocked due to access forbidden (403) The server refused Googlebot. Google: "Googlebot never provides credentials, so your server is returning this error incorrectly." Google's fix is to admit signed-out users or allow Googlebot without authentication ( Google: Page indexing report ). If a firewall or bot protection rule sends the 403, let Googlebot through for the pages you want indexed. Google publishes how to verify that a request comes from Googlebot before you allow it. URL blocked due to other 4xx issue Any other 400-level error. Google's advice is to debug the page with the URL Inspection tool ( Google: Page indexing report ). Reasons where Google has not indexed the page yet Google crawled the page and left it out, or found the address and has not crawled it yet. In the report, Google lists the source of "Crawled - currently not indexed" as Google systems, as our own capture above shows. Crawled - currently not indexed Google fetched the page and did not index it. Google: "Google doesn't index all pages across the web, so this is a common message for many sites and doesn't necessarily indicate an issue that needs attention." For important pages, inspect them with URL Inspection ( Google: Page indexing report ). The live test cannot check this status, so a clean live test does not settle it ( Google: URL Inspection tool ). Asking again does not speed it up. Requesting a crawl "does not guarantee that inclusion in search results will happen instantly or even at all", and asking for the same address several times "won't get it crawled any faster" ( Google: Ask Google to recrawl your URLs ). Our advice: work on the page itself, on what it gives a reader that the pages already indexed do not. Discovered - currently not indexed Google knows the address but has not fetched it yet. Google: "Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report." ( Google: Page indexing report ). Google gives no fix for this status. Its general advice for pages it has not reached: a page must be "linked from a known page, or from a sitemap", so "make sure they are linked properly" ( Google: Page indexing report ). Reasons about duplicates and canonicals Google found the same content at more than one address and indexed one of them. Google: "Google does not serve duplicate pages" ( Google: Page indexing report ). Duplicate without user-selected canonical Google sees the page as a copy of another one, and the page does not name its preferred version. Google: "This is not an error, but is working as intended." If you prefer a particular version, add a canonical tag that names it ( Google: Page indexing report ). Google's help names filter and sort parameters such as sort=price as a common source of duplicates. Duplicate, Google chose different canonical than user Your page names itself as the canonical, but Google picked another address. In URL Inspection, compare User-declared canonical with Google-selected canonical ( Google: Page indexing report ). A canonical tag is "a hint, not a rule" ( Google: Canonicalization ). Our advice: if the two pages really are the same, let Google's choice stand or point everything at one address (links, sitemap, redirects). If they are different, make the difference visible in the content. Duplicate, submitted URL not selected as canonical Google lists this status in its URL Inspection help, among the canonical reasons the live test cannot check: "Information about Google's canonical choice is determined at indexing time" ( Google: URL Inspection tool ). The Page indexing help gives it no entry of its own. Read it the same way as the status above: inspect the address and compare the two canonical fields. Indexed, with a warning These pages are in Google, with a problem Google wants you to see. Indexed, though blocked by robots.txt Google: "The page was indexed despite being blocked by your website's robots.txt file." To keep it out, remove the robots.txt block and use noindex. If the block is a mistake, unblock the page ( Google: Page indexing report ). Page indexed without content Google indexed the address but "could not read the content". Google adds: "This is not a case of robots.txt blocking." Inspect the page in URL Inspection ( Google: Page indexing report ). After the fix Test one address live. In URL Inspection, click Test live URL. A pass means Google can reach the page and is allowed to index it, nothing more: "There is no definitive test that can guarantee whether your page will be included in the Google index" ( Google: URL Inspection tool ). Request indexing for that page if it matters. Google: "Submitting a request does not guarantee that the page will appear in the Google Index." There is a daily limit, so for many pages submit a sitemap instead ( Google: URL Inspection tool ). Click Validate fix on the reason's page in the Page indexing report; the row's Validation column then leaves Not Started. Click it once: "Do not click Validate fix again until validation has succeeded or failed." Google says it "typically takes up to about two weeks" and can take much longer ( Google: Page indexing report ). Validation is optional. Google updates the counts whenever it recrawls a page with a known issue ( Google: Page indexing report ). Questions people ask How long does Google take to index a new page? Google gives ranges, not a promise. Its URL Inspection help says indexing "can take up to a week or two" and "typically takes only a day or so, but can take much longer in some cases" ( Google: URL Inspection tool ), and crawling "can take anywhere from a few days to a few weeks" ( Google: Ask Google to recrawl your URLs ). Will Google index every page of my site? No. Google: "Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials" ( Google: How Search works ). Its Search Console help adds: "Most sites have at least a few unindexed pages, but all of your important pages should be indexed" ( Google: Page indexing report ). URL Inspection shows no problem, but the report still shows an error. Why? Google's answer: "You might have fixed the error after the URL was last crawled by Google." Compare the crawl date of the URL with the date of your fix. Google adds that the live test does not check every issue in the report: "Most notably, duplicate or canonical conditions are not tested in the live test" ( Google: Page indexing report ). The other way round, a page that URL Inspection shows as indexed but the report does not list yet "was probably indexed recently, and will appear in the Page indexing report soon" ( Google: Page indexing report ). Does an indexed page show up in search? Not always. Google: "Just because a page is indexed doesn't guarantee that it will show up in your search results." Results change with each searcher's history, location and other factors ( Google: Page indexing report ). Do I need Bing as well? For Bing, our guide shows how to set up Bing Webmaster Tools for a new site. Sources Google Search Console Help: Page indexing report Google Search Console Help: URL Inspection tool Google Search Central: Introduction to robots.txt Google Search Central: Block Search indexing with noindex Google Search Central: Canonicalization Google Search Central: Ask Google to recrawl your URLs Google Search Central: How Google Search works Google Search Central: JavaScript SEO basics Google: Verify Google requests Check what Googlebot meets on your site Type your domain. We read your robots.txt and your sitemap, then request your home page once as Googlebot, Bingbot and the AI search crawlers. You get one line per engine and the next step. Check my site Result in seconds. No account, no card. Keep reading Is my site findable by Google, Bing and ChatGPT? Free check How to set up Bing Webmaster Tools for a new site Technical SEO checklist Googlebot: what it reads and how to verify it ## Technical SEO checklist for AI visibility: 10 shop checks Source: https://citedby.eu/guides/technical-seo-checklist Guides / Guide Technical SEO checklist for AI visibility: 10 checks for your online shop By CitedBy · Page updated 26 October 2026. 7 sources · method Explanation written with AI from the official source cited on this page. Short answer A technical SEO checklist for AI visibility has ten checks: robots.txt, HTTP status, sitemap, canonical, hreflang, title and H1, JSON-LD, text in the raw HTML, internal links and, optionally, llms.txt. Each one can be tested by hand in a browser. Our audit runs them on every page up to your plan's page limit, from €9, with the exact fix. On this page What does a technical check for AI visibility cover? What are the 10 checks? Which checks matter most? How do I test without any tools? What does the audit add? Questions people ask Sources What does a technical check for AI visibility cover? AI assistants and search engines can only quote what their crawlers can fetch and read. A page that returns an error, is blocked in robots.txt or shows its text only after JavaScript runs may look fine to you and empty to a crawler. The checks below are ordered from "can it get in" to "can it read the page" to "can it trust the facts". You need a browser and a few minutes per check. Test your home page, one category page and one product page first, because problems usually repeat across a template. What are the 10 checks? Check robots.txt and AI crawlers. Open yourshop.com/robots.txt. Look for Disallow: / under User-agent: * and for groups that name GPTBot, OAI-SearchBot or PerplexityBot. OpenAI documents its crawlers and says robots.txt controls them. Check the HTTP status of key pages. Your home, category, product and policy pages should answer 200. Run curl -sI https://yourshop.com/product-page and read the first line. Redirect chains, 403, 404 and 5xx answers all cost you. Check that sitemap.xml exists. Open yourshop.com/sitemap.xml. It should list your product and category URLs, and robots.txt can point to it with a Sitemap: line. Check the canonical tag. Open a product page, view the source and search for rel="canonical". It should point to the clean address of that same page, not to the home page or a filtered variant. Check hreflang if you sell in several languages. Each language version should list its siblings and itself. Google's documentation asks for the annotations to be reciprocal. Check the title and H1. Each page needs one title and one H1 that say what the page is. Duplicate titles across hundreds of products are common after a theme or import change. Check JSON-LD. Search the source for application/ld+json. A product page should carry Product data with name, price, currency and availability, and the home page should carry Organization data. The values must match what the page shows. Check that the text is in the raw HTML. Open the page source (not the inspector) and search for a sentence from the product description. If it is missing, the text is added by JavaScript and many crawlers will not see it. Check broken internal links. Click through your menu, a category and a product. A crawler that hits a 404 from your own navigation stops following that path. Fix or remove the link. Decide on llms.txt last. It is optional. No major assistant has confirmed that it reads the file, so do it only after the nine checks above. See does llms.txt work. Which checks matter most? Check If it fails Priority robots.txt Crawlers are told to stay out First HTTP status Pages cannot be fetched at all First Raw HTML text The crawler sees an empty page First Sitemap Pages are found more slowly Second Canonical Duplicates compete with each other Second JSON-LD Price and stock are guessed from text Second Title and H1 Pages look alike to a reader Second Hreflang Wrong language version gets shown If multilingual Internal links Crawl paths end early Third llms.txt Nothing measurable so far Optional How do I test without any tools? A browser is enough for most of it. The "view source" page (Ctrl+U, or Cmd+Option+U on a Mac) answers checks 4, 6, 7 and 8. The address bar answers checks 1 and 3. A single curl line answers check 2. One limit: a test from your own computer can differ from what a crawler gets. A CDN or firewall may refuse a crawler by its name or address while letting you in. If robots.txt looks fine and a crawler is still locked out, see can ChatGPT read my website. What does the audit add? By hand you check a few pages. The CitedBy audit crawls your shop up to your plan's page limit, runs these checks on each one and gives the exact fix for each issue. It starts at €9, paid once, with a 14-day money-back guarantee. The sample audit shows the format. Questions people ask How often should I run these checks? After every theme change, app install, migration or CDN setting change, and once a quarter otherwise. Most breakage appears right after a change. Do I need all ten checks to be recommended by AI assistants? No. Passing them gets your pages read, and assistants also weigh what other sites say about you when they choose what to mention. Failing the first three can keep you out entirely. Can I fix these myself on a hosted shop platform? Often yes, but the place to do it differs by platform. See the checklist by shop platform. Sources OpenAI: crawlers and user agents Google Search Central: robots.txt introduction Google Search Central: sitemaps overview Google Search Central: canonical URLs Google Search Central: localized versions (hreflang) Google Search Central: product structured data Google Search Central: JavaScript SEO basics Keep reading Can ChatGPT read my website? Does llms.txt work? AI visibility checklist by shop platform ## Amazonbot, Amzn-SearchBot and Amzn-User: three Amazon agents Source: https://citedby.eu/bots/amazonbot AI crawlers / Amazonbot Amazonbot, Amzn-SearchBot and Amzn-User: three Amazon agents, three rules By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Amazonbot is used to improve Amazon's products and services and may be used to train Amazon AI models. Amazon's page also covers Amzn-SearchBot, which makes content eligible for search experiences such as Alexa, and Amzn-User, which may not follow all robots.txt directives. Each setting is independent. Operator Amazon Job Product improvement Obeys robots.txt Yes Address list Published On this page What Amazonbot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with DuckAssistBot Check your own site Questions people ask Sources What Amazonbot is Amazon's page covers three user agents and says each setting is independent of the others and can take about 24 hours to take effect: Amazonbot, Amzn-SearchBot and Amzn-User. Only the first is described as possibly training Amazon AI models. Fact Amazonbot Operator Amazon Job Product improvement, may train Amazon AI models robots.txt token Amazonbot Follows robots.txt Yes, changes take about 24 hours Published address list Published on Amazon's page ( developer.amazon.com/amazonbot/ip-addresses ) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models.” Source: Amazon: About Amazonbot, read on 21 October 2026. What happens if you block it Amzn-SearchBot is for search experiences such as Alexa: allow it and the content is eligible to appear there. Amzn-User answers live Alexa queries that need up-to-date information; Amazon says it may not follow all robots.txt directives and does not crawl for generative AI training. A robots.txt that lists only Amazonbot therefore does not describe your policy for the other two. The robots.txt lines To allow it: User-agent: Amazonbot Allow: / User-agent: Amzn-SearchBot Allow: / To block it: User-agent: Amazonbot Disallow: / The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one Amazon publishes the addresses for Amazonbot and, on separate pages, for Amzn-SearchBot and for live fetches; the lists are HTML pages, not a JSON file, so we do not state a count. Amazon also says that if robots.txt does not mention Amzn-SearchBot but allows other search bots, Amzn-SearchBot crawls in accordance with that file. Compare with DuckAssistBot DuckAssistBot : Another answer-engine fetcher with a stated opt-out and a 72-hour delay. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does Amzn-User follow robots.txt? Not always. Amazon says that because its actions can be initiated by a user, Amzn-User may not follow all robots.txt directives. It also says it does not crawl content for generative AI model training. Does blocking Amazonbot block Alexa answers? Not by that line alone. Amazon describes Amzn-SearchBot and Amzn-User as separate agents with independent settings. Sources Amazon: About Amazonbot developer.amazon.com/amazonbot/ip-addresses, read 21 October 2026 Keep reading All AI crawlers compared DuckAssistBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## Amzn-SearchBot: what it does and what blocking it changes Source: https://citedby.eu/bots/amzn-searchbot AI crawlers / Amzn-SearchBot Amzn-SearchBot: what it does and what blocking it changes By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Amzn-SearchBot is an Amazon crawler used to improve search experiences such as Alexa. Amazon says allowing it makes your content eligible to appear there, and that it does not crawl for generative AI model training. Its setting is independent of Amazonbot and Amzn-User, and a change can take about 24 hours. Operator Amazon Job Search Obeys robots.txt Yes Address list Listed on its page On this page What Amzn-SearchBot is What the vendor says What happens if you block Amzn-SearchBot The robots.txt lines How to tell a real request from a fake one Compare with Amazonbot Your own site Questions people ask Sources What Amzn-SearchBot is Amzn-SearchBot is an Amazon crawler used to improve search experiences in Amazon products and services. Amazon says that allowing it makes your content eligible to appear in search experiences such as Alexa, and that it does not crawl content for generative AI model training. It is one of three Amazon user agents (Amazonbot, Amzn-SearchBot, Amzn-User) that Amazon documents on one page. Fact Amzn-SearchBot Operator Amazon Job Search robots.txt token Amzn-SearchBot Follows robots.txt Yes, changes may take about 24 hours. IP addresses Listed on the vendor's page ( developer.amazon.com/amazonbot/searchbot-ip-addresses ) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “If robots.txt files don’t mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots.” Source: Amazon: About Amazonbot, read on 24 October 2026. What happens if you block Amzn-SearchBot Amazon says automated crawling from its listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and allow/disallow directives. Amazon says each user agent setting is independent of the others and may take about 24 hours to take effect. If robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the directives given to those other search bots. The robots.txt lines To allow Amzn-SearchBot: User-agent: Amzn-SearchBot Allow: / To block Amzn-SearchBot: User-agent: Amzn-SearchBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one Amazon publishes the Amzn-SearchBot IP addresses on a web page that shows the list as JSON, and links it from the Amazonbot documentation page. Compare with Amazonbot Amazonbot : Amazonbot is used to improve Amazon products and services and may be used to train Amazon AI models, while Amzn-SearchBot is for search experiences and Amazon says it does not crawl for generative AI model training. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Amzn-SearchBot is not one of them: read your robots.txt for a Amzn-SearchBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Amzn-SearchBot follow robots.txt? Yes, changes may take about 24 hours. Give Amzn-SearchBot its own User-agent group in robots.txt: the operator's other crawlers have their own settings. How do I know Amzn-SearchBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Amazon: About Amazonbot Amazon: Amzn-SearchBot IP addresses (read 2026-10-24) Keep reading All AI crawlers compared Amazonbot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Amzn-User: what it does and what blocking it changes Source: https://citedby.eu/bots/amzn-user AI crawlers / Amzn-User Amzn-User: what it does and what blocking it changes By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Amzn-User is the Amazon user agent that fetches live pages when a customer asks Alexa something that needs current information. Amazon says it does not crawl for generative AI training and may not follow all robots.txt directives, because the action is user-initiated. Amazonbot and Amzn-SearchBot are separate settings. Operator Amazon Job User-triggered fetch Obeys robots.txt Not always Address list Listed on its page On this page What Amzn-User is What the vendor says What happens if you block Amzn-User The robots.txt lines How to tell a real request from a fake one Compare with Amzn-SearchBot Your own site Questions people ask Sources What Amzn-User is Amzn-User is an Amazon user agent that supports user actions, such as responding to Alexa queries that require up-to-date information. Amazon says that when a customer asks a question, Amzn-User may fetch live information from the web to provide accurate answers on the user's behalf. Amazon says it does not crawl content for generative AI model training. Fact Amzn-User Operator Amazon Job User-triggered fetch robots.txt token Amzn-User Follows robots.txt Not always. Amazon says Amzn-User may not follow all robots.txt directives because actions can be user-initiated. IP addresses Listed on the vendor's page ( developer.amazon.com/amazonbot/live-ip-addresses ) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives.” Source: Amazon: About Amazonbot, read on 24 October 2026. What happens if you block Amzn-User Amazon says Amzn-User may not follow all robots.txt directives because its actions can be initiated by a user. Amazon also says each of its user agent settings (Amazonbot, Amzn-SearchBot, Amzn-User) is independent of the others and may take about 24 hours to reflect changes. The robots.txt lines To allow Amzn-User: User-agent: Amzn-User Allow: / To block Amzn-User: User-agent: Amzn-User Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one Amazon publishes the Amzn-User IP addresses on a web page that shows the list as JSON, and links it from the Amazonbot documentation page. Compare with Amzn-SearchBot Amzn-SearchBot : Amzn-SearchBot is an automated crawler for Amazon search experiences that Amazon says respects robots.txt, while Amzn-User fetches live information when a customer asks and may not follow all directives. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Amzn-User is not one of them: read your robots.txt for a Amzn-User group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Amzn-User follow robots.txt? Not always. Amazon says Amzn-User may not follow all robots.txt directives because actions can be user-initiated. A rule for Amzn-User in robots.txt may therefore not stop it. To refuse it for certain, use a firewall or CDN rule that matches both the user agent and the published addresses. How do I know Amzn-User is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Amazon: About Amazonbot Amazon: Amzn-User IP addresses (read 2026-10-24) Keep reading All AI crawlers compared Amzn-SearchBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Applebot: what it does and what blocking it changes Source: https://citedby.eu/bots/applebot AI crawlers / Applebot Applebot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Applebot is Apple's web crawler. Apple says its data powers search in Spotlight, Siri and Safari and may also help train Apple's foundation models. Disallowing Applebot-Extended opts content out of training only. If robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Operator Apple Job Several jobs Obeys robots.txt Yes Address list 24 ranges On this page What Applebot is What the vendor says What happens if you block Applebot The robots.txt lines How to tell a real request from a fake one Compare with Applebot-Extended Your own site Questions people ask Sources What Applebot is Applebot is Apple's web crawler. Apple says its data powers search technology in Spotlight, Siri and Safari, and may also be used to help train Apple foundation models and to provide context when AI models generate output in Apple products. Apple also documents a separate Applebot-Extended token and an iTMS user agent for Apple Podcasts. Fact Applebot Operator Apple Job Search (crawled data may also help train Apple models) robots.txt token Applebot Follows robots.txt Yes. Apple says Applebot follows robots.txt in general search crawls that are targeted at Applebot. IP addresses 24 ranges ( search.developer.apple.com/applebot.json, file dated 15 September 2026) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot.” Source: Apple: About Applebot, read on 24 October 2026. What happens if you block Applebot Apple says disallowing Applebot-Extended opts content out of training Apple's generative foundation models, but the site may still be crawled by Applebot and stay discoverable through Spotlight, Siri and Safari. A Disallow for Applebot itself applies to general search crawls targeted at Applebot, and Apple says that if robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot instructions. Apple says the nosnippet meta tag opts content out of broad world knowledge answers. The robots.txt lines To allow Applebot: User-agent: Applebot Allow: / To block Applebot: User-agent: Applebot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) How to tell a real request from a fake one The file at search.developer.apple.com/applebot.json held 24 ranges on 24 October 2026, file dated 15 September 2026. Apple says Applebot traffic is generally identified by reverse DNS in the *.applebot.apple.com domain, or by matching the IP address with a CIDR prefix in its published JSON file. Compare with Applebot-Extended Applebot-Extended : Applebot is the crawler that fetches pages for Apple search features, while Applebot-Extended is the robots.txt token that controls whether crawled content is used to train Apple generative models. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Applebot is not one of them: read your robots.txt for a Applebot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Applebot follow robots.txt? Yes. Apple says Applebot follows robots.txt in general search crawls that are targeted at Applebot. Give Applebot its own User-agent group in robots.txt: the operator's other crawlers have their own settings. How do I know Applebot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Apple: About Applebot Apple: Applebot IP CIDRs JSON (read 2026-10-24) Keep reading All AI crawlers compared Applebot-Extended AI crawler settings by platform Can ChatGPT read my website? What decides it ## Applebot-Extended: opt out of Apple model training, keep search Source: https://citedby.eu/bots/applebot-extended AI crawlers / Applebot-Extended Applebot-Extended: opting out of Apple's model training without leaving search By CitedBy · Page updated 22 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Applebot-Extended is a robots.txt token. Apple says publishers can opt out of having their content used to train generative foundation models by disallowing it. Applebot may still crawl the page, so the content stays discoverable in Spotlight, Siri and Safari. It makes no requests of its own. Operator Apple Job Control token Obeys robots.txt Read from robots.txt only Address list None On this page What Applebot-Extended is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Applebot Check your own site Questions people ask Sources What Applebot-Extended is Apple's page describes Applebot-Extended as the way to keep content out of training for foundation models behind Apple Intelligence, services and developer tools. Like Google-Extended it is a control token: you will not find it in a log. Fact Applebot-Extended Operator Apple Job Control token (no requests of its own) robots.txt token Applebot-Extended Follows robots.txt Read from robots.txt only Published address list None: it never fetches Vendor page read 21 October 2026 What the vendor says In the vendor's words: “Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file.” Source: Apple Support: About Applebot, read on 21 October 2026. What happens if you block it Apple spells out the limit: if you disallow Applebot-Extended and also tag content with the nosnippet meta tag, your instructions may still allow Applebot to crawl the page, and the content remains discoverable through Spotlight, Siri and Safari. The same page says publishers can opt out of use in broad world-knowledge answers by applying this robots.txt control. The robots.txt lines To allow it: User-agent: Applebot-Extended Allow: / To block it: User-agent: Applebot-Extended Disallow: / How to tell a real request from a fake one No network check exists for a token that sends no requests. Read robots.txt, on each host, and check Applebot separately, because the two tokens are independent and a single User-agent: Applebot-Extended group says nothing about Applebot. Compare with Applebot Applebot : The crawler itself: allowing it keeps you in Spotlight, Siri and Safari search. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Why does Applebot-Extended not show up in my server logs? It sends no requests. Apple describes it as a robots.txt control for whether crawled content is used for training. Does Disallow Applebot-Extended remove me from Siri? Apple says content stays discoverable through Spotlight, Siri and Safari because Applebot may still crawl it. Sources Apple Support: About Applebot Keep reading All AI crawlers compared Is GPTBot blocked on your site? Can ChatGPT read my website? ## Baiduspider: what it does and what blocking it changes Source: https://citedby.eu/bots/baiduspider AI crawlers / Baiduspider Baiduspider: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Baiduspider is the crawler of the Baidu search engine. Baidu says it follows the robots.txt protocol and that blocking Baiduspider blocks all Baidu user agents, so pages will not be found in Baidu results. Baiduspider-cpro and Baiduspider-ads work on customer agreements and do not follow robots.txt records. Operator Baidu Job Search Obeys robots.txt Yes Address list Reverse DNS On this page What Baiduspider is What the vendor says What happens if you block Baiduspider The robots.txt lines How to tell a real request from a fake one Compare with Bingbot Your own site Questions people ask Sources What Baiduspider is Baiduspider is the crawler of the Baidu search engine; it visits pages and builds them into the Baidu index. Baidu uses different user agents for different products, for example Baiduspider for PC and mobile search and Baiduspider-image, Baiduspider-video and Baiduspider-news for those searches. Fact Baiduspider Operator Baidu Job Search robots.txt token Baiduspider Follows robots.txt Yes for Baiduspider; Baiduspider-cpro and Baiduspider-ads do not follow robots.txt records. IP addresses Reverse DNS check Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Baiduspider is Baidu search engine program which is used to visit pages on the internet and build information into Baidu index.” Source: Baidu Help Center: FAQs of Baiduspider, read on 24 October 2026. What happens if you block Baiduspider Baidu says Baiduspider works on the robots.txt protocol, and that blocking it means your pages will not be found in Baidu search results or in any other results provided by Baidu. You can set different rules for different Baidu user agents, and blocking Baiduspider blocks all Baidu user agents. Baiduspider-cpro and Baiduspider-ads work on agreements with customers and do not follow robots.txt records. The robots.txt lines To allow Baiduspider: User-agent: Baiduspider Allow: / To block Baiduspider: User-agent: Baiduspider Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Baiduspider How to tell a real request from a fake one Baidu recommends a reverse DNS lookup on the visiting IP; the host name of Baiduspider is *.baidu.com or *.baidu.jp, and other host names are fake. Compare with Bingbot Bingbot : Both crawl for a search engine; Baidu documents several product-specific user agents (image, video, news, ads) under the Baiduspider name and verifies by reverse DNS host name. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Baiduspider is not one of them: read your robots.txt for a Baiduspider group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Baiduspider follow robots.txt? Yes for Baiduspider; Baiduspider-cpro and Baiduspider-ads do not follow robots.txt records. How do I know Baiduspider is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Baidu Help Center: FAQs of Baiduspider Baidu Help Center: How to block the crawling (robots.txt) Keep reading All AI crawlers compared Bingbot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Bingbot: Microsoft's crawler, five user agents, how to verify Source: https://citedby.eu/bots/bingbot AI crawlers / Bingbot Bingbot: Microsoft's crawler, five user agents, how to verify By CitedBy · Page updated 30 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Bingbot is Microsoft's standard crawler and handles most of Bing's crawling. Bing's help page lists five crawlers: Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview. You control them through robots.txt, and crawl rate through the Crawl control tool in Bing Webmaster Tools, not through robots.txt. Operator Microsoft Job Search crawler Obeys robots.txt Yes Address list Verify tool, no list read On this page What Bingbot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Check your own site Questions people ask Sources What Bingbot is Bing's page is titled "Overview of Bing crawlers (user agents)" and says Bing operates these main crawlers today: Bingbot (the standard crawler), AdIdxBot (used by Bing Ads, which crawls ads and follows their websites for quality control), BingPreview (page snapshots for Bing), MicrosoftPreview (snapshots for Microsoft products) and BingVideoPreview (video previews, in Bing only). Bingbot, AdIdxBot, BingPreview and BingVideoPreview have desktop and mobile variants. The page does not name a separate AI crawler. The same help section includes "How To Permanently Remove a URL or Page from Bing or Copilot", which is the page to read if the goal is removal rather than access. Fact Bingbot Operator Microsoft Job Search crawler robots.txt token bingbot Follows robots.txt Yes Published address list Verify tool, no list read Vendor page read 21 October 2026 What the vendor says In the vendor's words: “Bingbot is our standard crawler and handles most of our crawling needs each day.” Source: Microsoft Bing Webmaster help: Which Crawlers Does Bing Use?, read on 21 October 2026. What happens if you block it Bing's page does not say what a block costs; this is our reading, not Bing's. Bingbot is the crawler that feeds Bing, so disallowing it in robots.txt stops Bing crawling those pages, and anything that relies on Bing's index loses them too. Bing gives you two controls: robots.txt tells its crawlers how to interact with your site, and Bing Webmaster Tools has a Crawl control tool that sets the crawl rate by the hour. The robots.txt lines To allow it: User-agent: bingbot Allow: / To block it: User-agent: bingbot Disallow: / The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one Bing says user agent strings are easy to spoof, so not every request with them comes from a real Bing crawler. To check the address, use the Verify Bingbot tool, or one of the other verification methods Bing describes. In the user-agent string, W.X.Y.Z stands for the Microsoft Edge version Bing renders with, which Bing says it updates regularly. We did not read an IP list on that page, so we state no count. Compare with Googlebot Googlebot : The other main search crawler: a different token and different verification. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Is there a separate Bing AI crawler? Bing's page lists five crawlers, and none of them is an AI-specific one. Wix's example list names BingAI, which is not on Bing's list. How do I slow Bingbot down? Bing says to use the Crawl control tool in Bing Webmaster Tools, which sets the crawl rate by the hour. How do I know a request is really Bingbot? Do not trust the user agent alone. Bing says it can be faked and points to its Verify Bingbot tool. Sources Microsoft Bing Webmaster help: Which Crawlers Does Bing Use?, read 21 October 2026 Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? ## CCBot: what it does and what blocking it changes Source: https://citedby.eu/bots/ccbot AI crawlers / CCBot CCBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data anyone can access and analyze. Common Crawl says User-agent: CCBot with Disallow: / stops it crawling a site. It also warns that other crawlers falsely identify themselves as CCBot, so verify requests. Operator Common Crawl Job Several jobs Obeys robots.txt Yes Address list Reverse DNS On this page What CCBot is What the vendor says What happens if you block CCBot The robots.txt lines How to tell a real request from a fake one Compare with GPTBot Your own site Questions people ask Sources What CCBot is CCBot is the crawler of Common Crawl, a non-profit foundation that produces and maintains an open repository of web crawl data that anyone can access and analyze. The vendor page says the crawl data is meant to be freely available to organizations, academia and non-profits. Fact CCBot Operator Common Crawl Job Open web crawl archive robots.txt token CCBot Follows robots.txt Yes, via a robots.txt rule for CCBot. IP addresses Reverse DNS check Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “To prevent Common Crawl from crawling your website, include the following in your robots.txt” Source: Common Crawl: CCBot, read on 24 October 2026. What happens if you block CCBot Common Crawl says that adding User-agent: CCBot with Disallow: / to robots.txt prevents Common Crawl from crawling the site. The page also says it is aware of crawlers falsely identifying themselves as CCBot and recommends verifying user agent strings. The robots.txt lines To allow CCBot: User-agent: CCBot Allow: / To block CCBot: User-agent: CCBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: CCBot/2.0 (https://commoncrawl.org/faq/) How to tell a real request from a fake one Common Crawl says CCBot runs on dedicated IP address ranges with reverse DNS (not for IPv6), so a logged request can be checked with a reverse lookup that resolves to a crawl.commoncrawl.org hostname; the ranges are also published as JSON at https://index.commoncrawl.org/ccbot.json. Compare with GPTBot GPTBot : CCBot builds a public crawl archive that others can use, while GPTBot collects content for OpenAI's own purposes. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). CCBot is not one of them: read your robots.txt for a CCBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does CCBot follow robots.txt? Yes, via a robots.txt rule for CCBot. How do I know CCBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Common Crawl: CCBot (read 2026-10-24) Keep reading All AI crawlers compared GPTBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## ChatGPT-User: the fetcher that opens your page when someone asks Source: https://citedby.eu/bots/chatgpt-user AI crawlers / ChatGPT-User ChatGPT-User: the fetcher that opens your page when someone asks By CitedBy · Page updated 26 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer ChatGPT-User visits a web page when a person asks ChatGPT or a custom GPT a question that needs it. OpenAI says it is not used to crawl automatically and that, because a user starts the request, robots.txt rules may not apply. OpenAI lists 234 IP ranges for it. Operator OpenAI Job User-triggered fetch Obeys robots.txt Not always Address list 234 ranges On this page What ChatGPT-User is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with OAI-SearchBot Check your own site Questions people ask Sources What ChatGPT-User is Unlike the other OpenAI robots, ChatGPT-User does not roam the web by itself. OpenAI describes it as the agent used for certain user actions in ChatGPT and custom GPTs, including pages a custom GPT reaches through GPT Actions. A person typed something, and a request to your server followed. Fact ChatGPT-User Operator OpenAI Job User-triggered fetch robots.txt token ChatGPT-User Follows robots.txt Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User, because its actions are initiated by a user. Published address list 234 ranges ( openai.com/chatgpt-user.json, file dated 2026-10-20) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.” Source: OpenAI: Overview of OpenAI Crawlers, read on 21 October 2026. What happens if you block it Because the visit is user-initiated, OpenAI warns that robots.txt rules may not apply, so a Disallow line for ChatGPT-User is a request that user-initiated visits may not follow. OpenAI also says ChatGPT-User is not used to decide whether content appears in search and sends search opt-outs to OAI-SearchBot. If you need to stop these requests, the control is on your server or CDN, not in robots.txt alone. The robots.txt lines To allow it: User-agent: ChatGPT-User Allow: / To block it: User-agent: ChatGPT-User Disallow: / # OpenAI says robots.txt rules may not apply to this agent The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot How to tell a real request from a fake one Its address list is the longest of the three OpenAI files we counted: openai.com/chatgpt-user.json held 234 prefixes on 21 October 2026, creation time 2026-10-20, the day before this page. A list that long is a reason to fetch it again when you check a log line, not to copy it once into a firewall. Compare with OAI-SearchBot OAI-SearchBot : OAI-SearchBot is the search crawler and does obey robots.txt. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does ChatGPT-User obey robots.txt? Not always. OpenAI says robots.txt rules may not apply to it because a person starts each request. Treat a robots.txt block as a request that OpenAI is not committed to honour here. Is ChatGPT-User the same as GPTBot? No. GPTBot crawls content that may be used for training. ChatGPT-User only fetches a page when a user action calls for it, and OpenAI says it is not used to crawl the web automatically. Sources OpenAI: Overview of OpenAI Crawlers openai.com/chatgpt-user.json, read 21 October 2026 Keep reading All AI crawlers compared OAI-SearchBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## Claude-SearchBot: Anthropic's search crawler and blocking it Source: https://citedby.eu/bots/claude-searchbot AI crawlers / Claude-SearchBot Claude-SearchBot: Anthropic's search crawler and what blocking changes By CitedBy · Page updated 26 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Claude-SearchBot is the Anthropic robot that navigates the web to improve search result quality. Anthropic says disabling it prevents indexing of your content for search optimization, which may reduce your visibility and accuracy in user search results. Its robots.txt token is Claude-SearchBot. Operator Anthropic Job Search Obeys robots.txt Yes Address list Anthropic points to a list On this page What Claude-SearchBot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with ClaudeBot Check your own site Questions people ask Sources What Claude-SearchBot is Claude-SearchBot is the indexing side of Anthropic's search. Anthropic describes it as analysing online content to improve relevance and accuracy of search responses. ClaudeBot collects for training and Claude-User answers a live question. Fact Claude-SearchBot Operator Anthropic Job Search robots.txt token Claude-SearchBot Follows robots.txt Yes, Anthropic says its robots respect robots.txt Published address list Anthropic points to a list but advises against IP blocking Vendor page read 21 October 2026 What the vendor says In the vendor's words: “Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.” Source: Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026), read on 21 October 2026. What happens if you block it Anthropic names two effects of disabling it: your content is not indexed for search optimization, and your site's visibility and accuracy in user search results may fall. The robots.txt lines To allow it: User-agent: Claude-SearchBot Allow: / To block it: User-agent: Claude-SearchBot Disallow: / How to tell a real request from a fake one The help article is dated 7 April 2026. It lists no user-agent string, so a log line that names Claude-SearchBot is a claim you cannot confirm from the article alone. Anthropic says traffic from an address on its linked list is Anthropic's, and it asks site owners who think a robot is misbehaving to write from an email on the domain concerned. Compare with ClaudeBot ClaudeBot : ClaudeBot is Anthropic's training crawler: Anthropic says a Disallow for it means future materials should be excluded from its AI model training datasets. Claude-SearchBot has its own token. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Do I have to allow Claude-SearchBot to be cited by Claude? Anthropic says that disabling it may reduce your visibility and accuracy in user search results. Which pages Claude cites in an answer is not described in that article. Where do I put the rule? In the robots.txt at the top level of each host. Anthropic says to do this for every subdomain. Sources Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026) Keep reading All AI crawlers compared ClaudeBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## Claude-User: fetches pages when someone asks Claude Source: https://citedby.eu/bots/claude-user AI crawlers / Claude-User Claude-User: the agent that fetches pages when someone asks Claude By CitedBy · Page updated 24 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Claude-User is the agent Claude may use to access a website when a person asks it a question. Anthropic says disabling it prevents retrieval of your content in response to a user query, which may reduce your visibility for user-directed web search. It is separate from ClaudeBot and Claude-SearchBot. Operator Anthropic Job User-triggered fetch Obeys robots.txt Yes Address list Anthropic points to a list On this page What Claude-User is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Claude-SearchBot Check your own site Questions people ask Sources What Claude-User is Of Anthropic's three robots, Claude-User is the one tied to a person's question. It does not collect training data ( ClaudeBot does) and it does not build the search index ( Claude-SearchBot does). Fact Claude-User Operator Anthropic Job User-triggered fetch robots.txt token Claude-User Follows robots.txt Yes, Anthropic says its robots respect robots.txt Published address list Anthropic points to a list but advises against IP blocking Vendor page read 21 October 2026 What the vendor says In the vendor's words: “Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners to control which sites can be accessed through these user-initiated requests. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.” Source: Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026), read on 21 October 2026. What happens if you block it This is where Anthropic's wording on cost is most direct: disabling Claude-User prevents retrieval "in response to a user query" and "may reduce your site's visibility for user-directed web search". A shop that blocks it can still be known to the model from other sources, but Claude cannot open the page when asked. The robots.txt lines To allow it: User-agent: Claude-User Allow: / To block it: User-agent: Claude-User Disallow: / How to tell a real request from a fake one Anthropic's help article does not print a user-agent string for the three robots, and it says its source IP addresses are on a list it links. Because Anthropic also says that blocking those addresses can stop its robots reading robots.txt, the article's advice is to control access in robots.txt rather than at the IP layer. Compare with Claude-SearchBot Claude-SearchBot : Search quality robot; its block affects indexing for search, not a user's live question. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask What is the difference between Claude-User and Claude-SearchBot? Claude-User fetches a page when a person asks Claude a question. Claude-SearchBot navigates the web to improve the quality of search results. They have separate robots.txt rules. Will blocking Claude-User hide my shop from Claude? Anthropic says it may reduce visibility for user-directed web search. It does not say it removes you from everything Claude knows. Sources Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026) Keep reading All AI crawlers compared Claude-SearchBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## ClaudeBot: Anthropic's training crawler, Crawl-delay and blocking Source: https://citedby.eu/bots/claudebot AI crawlers / ClaudeBot ClaudeBot: Anthropic's training crawler, Crawl-delay and blocking By CitedBy · Page updated 23 October 2026. method Explanation written with AI from the official source cited on this page. Short answer ClaudeBot is Anthropic's crawler for content that could contribute to model training. A robots.txt Disallow signals that a site's future materials should be excluded from training datasets. Anthropic honours Crawl-delay, and warns that blocking its IP addresses can stop it reading robots.txt, so use robots.txt instead. Operator Anthropic Job Training Obeys robots.txt Yes Address list Anthropic points to a list On this page What ClaudeBot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Claude-SearchBot Check your own site Questions people ask Sources What ClaudeBot is Anthropic names three robots in its help article: ClaudeBot for training data, Claude-User for pages fetched at a user's request, and Claude-SearchBot for search quality. ClaudeBot is the training one. The article is dated 7 April 2026. Fact ClaudeBot Operator Anthropic Job Training robots.txt token ClaudeBot Follows robots.txt Yes, it follows robots.txt and Crawl-delay Published address list Anthropic points to a list but advises against IP blocking Vendor page read 21 October 2026 What the vendor says In the vendor's words: “ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training. When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.” Source: Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026), read on 21 October 2026. What happens if you block it Anthropic says a restriction signals that the site's future materials should be excluded from its training datasets. The wording says "future materials"; it does not say what happens to content already collected. The other two Claude robots are separate, so blocking ClaudeBot leaves their rules untouched. The robots.txt lines To allow it: User-agent: ClaudeBot Allow: / To block it: User-agent: ClaudeBot Disallow: / How to tell a real request from a fake one Anthropic's article says its robots respect "do not crawl" signals in robots.txt and will not try to bypass CAPTCHAs. It supports the non-standard Crawl-delay directive, for example User-agent: ClaudeBot followed by Crawl-delay: 1. It also says that blocking by IP address may not give a lasting opt-out, because it stops the robot from reading robots.txt. Anthropic's article states that robots.txt must be set for every subdomain you want to opt out. Compare with Claude-SearchBot Claude-SearchBot : A separate search robot: blocking ClaudeBot does not touch it. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does ClaudeBot support Crawl-delay? Yes. Anthropic says it supports the non-standard Crawl-delay extension and gives Crawl-delay: 1 under User-agent: ClaudeBot as an example. Do I need a robots.txt file on every subdomain? Anthropic's article says to add the rule at the top level for every subdomain you want to opt out. A rule on example.com does not cover shop.example.com. Sources Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 April 2026) Keep reading All AI crawlers compared Claude-SearchBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## Diffbot: what it does and what blocking it changes Source: https://citedby.eu/bots/diffbot AI crawlers / Diffbot Diffbot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Diffbot documents two user agents. Diffbot crawls the web proactively for a general search engine, so sites can be discovered and cited from its Knowledge Graph. Diffbot-User fetches a URL for a person using its software. Diffbot says its crawls follow robots.txt by default, including disallow and crawl-delay. Operator Diffbot Job Search Obeys robots.txt Yes Address list None On this page What Diffbot is What the vendor says What happens if you block Diffbot The robots.txt lines How to tell a real request from a fake one Compare with Bingbot Your own site Questions people ask Sources What Diffbot is Diffbot documents two user agents. Diffbot is used for general, proactive web crawling for building a general search engine, so sites can be discovered and cited from the Diffbot Knowledge Graph and web search services. Diffbot-User is used for requests made on behalf of a human user browsing a URL with Diffbot software. Fact Diffbot Operator Diffbot Job Search (general web crawl) robots.txt token Diffbot Follows robots.txt Yes. Diffbot says it follows disallow and crawl-delay directives by default. IP addresses None published Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “By default Diffbot's web crawls adhere to a site’s robots.txt instructions, including the disallow and crawl-delay directives.” Source: Diffbot: Does Diffbot respect robots.txt?, read on 24 October 2026. What happens if you block Diffbot Diffbot says its web crawls follow robots.txt by default, including disallow and crawl-delay, and that a rule can be overridden in specific cases such as a partnership or agreement with the site. It lists Diffbot and Diffbot-User as separate user agents, and says users of its Extract and Crawlbot APIs define their own crawl parameters and can set their own User-Agent. The robots.txt lines To allow Diffbot: User-agent: Diffbot Allow: / To block Diffbot: User-agent: Diffbot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Diffbot How to tell a real request from a fake one The vendor page does not describe an IP list or verification method. Compare with Bingbot Bingbot : Diffbot is described by its operator as similar to Googlebot and Bing, a general crawler for a search index, and the vendor says it is not used for AI training. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Diffbot is not one of them: read your robots.txt for a Diffbot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Diffbot follow robots.txt? Yes. Diffbot says it follows disallow and crawl-delay directives by default. How do I know Diffbot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Diffbot: Does Diffbot respect robots.txt? (read 2026-10-24) Keep reading All AI crawlers compared Bingbot AI crawler settings by platform Can ChatGPT read my website? What decides it ## DuckAssistBot: DuckDuckGo's answer crawler and its 486 addresses Source: https://citedby.eu/bots/duckassistbot AI crawlers / DuckAssistBot DuckAssistBot: DuckDuckGo's real-time answer crawler and its 486 addresses By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer DuckAssistBot crawls pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo says the data is not used to train AI models. A robots.txt Disallow takes effect after 72 hours and does not affect organic search rankings. It lists 486 IP addresses. Operator DuckDuckGo Job Real-time fetch for AI-assisted answers Obeys robots.txt Yes Address list 486 addresses On this page What DuckAssistBot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Amazonbot Check your own site Questions people ask Sources What DuckAssistBot is DuckDuckGo runs two bots that people confuse. DuckDuckBot is the search crawler; DuckAssistBot is the one behind AI-assisted answers. DuckDuckGo's help page says the answers prominently cite their sources and that the crawled data is not used to train AI models. Fact DuckAssistBot Operator DuckDuckGo Job Real-time fetch for AI-assisted answers robots.txt token DuckAssistBot Follows robots.txt Yes, a change takes effect after 72 hours Published address list 486 addresses ( duckduckgo.com/duckassistbot.json, file dated 2026-09-01) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers, which prominently cite their sources. This data is not used in any way to train AI models.” Source: DuckDuckGo Help Pages: About DuckAssistBot, read on 21 October 2026. What happens if you block it DuckDuckGo says that once you disallow DuckAssistBot the change takes effect after 72 hours, and that opting out does not impact organic search rankings and will not affect whether sites appear in its search results. The cost of a block is therefore limited to the assisted answers and their citations. Opt-out problems go to the address the page gives, crawling@duckduckgo.com. The robots.txt lines To allow it: User-agent: DuckAssistBot Allow: / To block it: User-agent: DuckAssistBot Disallow: / The user-agent string the vendor shows: DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) How to tell a real request from a fake one DuckDuckGo lists the addresses in the help article and as JSON at duckduckgo.com/duckassistbot.json. The file held 486 entries on 21 October 2026, creation time 2026-09-01. The DuckDuckBot file holds the same 486 entries, and both hold single addresses where most operators publish CIDR ranges. The user agent appears as DuckAssistBot/1.2. Compare with Amazonbot Amazonbot : Amazon's family of three agents, one of which may ignore robots.txt. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask How long does a DuckAssistBot opt-out take? DuckDuckGo says the change takes effect after 72 hours. Will blocking DuckAssistBot drop me from DuckDuckGo search? DuckDuckGo says it does not impact organic search rankings and does not affect whether sites appear in its search results. Sources DuckDuckGo Help Pages: About DuckAssistBot duckduckgo.com/duckassistbot.json, read 21 October 2026 Keep reading All AI crawlers compared Amazonbot Is GPTBot blocked on your site? Can ChatGPT read my website? ## DuckDuckBot: what it does and what blocking it changes Source: https://citedby.eu/bots/duckduckbot AI crawlers / DuckDuckBot DuckDuckBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer DuckDuckBot is the web crawler for DuckDuckGo, whose stated job is to improve DuckDuckGo's search results. DuckDuckGo says it respects WWW::RobotRules and publishes its IP addresses as a list in JSON form. DuckAssistBot, the fetcher for AI-assisted answers, is documented on a separate DuckDuckGo help page. Operator DuckDuckGo Job Search Obeys robots.txt Yes Address list 486 addresses On this page What DuckDuckBot is What the vendor says What happens if you block DuckDuckBot The robots.txt lines How to tell a real request from a fake one Compare with DuckAssistBot Your own site Questions people ask Sources What DuckDuckBot is DuckDuckBot is the web crawler for DuckDuckGo. DuckDuckGo says its job is to improve DuckDuckGo's search results. DuckDuckGo's help pages list DuckAssistBot on a separate page. Fact DuckDuckBot Operator DuckDuckGo Job Search robots.txt token DuckDuckBot Follows robots.txt Yes. DuckDuckGo says DuckDuckBot respects WWW::RobotRules and publishes no timing note for changes. IP addresses 486 addresses ( duckduckgo.com/duckduckbot.json, file dated 1 September 2026) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “DuckDuckBot’s job is to constantly improve our search results and offer users the best and most secure search experience possible.” Source: DuckDuckGo Help Pages: Is DuckDuckBot related to DuckDuckGo?, read on 24 October 2026. What happens if you block DuckDuckBot DuckDuckGo says DuckDuckBot respects WWW::RobotRules, so a Disallow rule addressed to DuckDuckBot in robots.txt applies to this crawler. The page says nothing about what blocking it changes in DuckDuckGo results, and nothing about DuckAssistBot. The robots.txt lines To allow DuckDuckBot: User-agent: DuckDuckBot Allow: / To block DuckDuckBot: User-agent: DuckDuckBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) How to tell a real request from a fake one The file at duckduckgo.com/duckduckbot.json held 486 addresses on 24 October 2026, file dated 1 September 2026. DuckDuckGo says the crawler originates from the IP addresses listed on its help page, also available as JSON at duckduckgo.com/duckduckbot.json. Compare with DuckAssistBot DuckAssistBot : DuckDuckBot is the crawler for DuckDuckGo search results, while DuckAssistBot is documented by DuckDuckGo on its own help page. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). DuckDuckBot is not one of them: read your robots.txt for a DuckDuckBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does DuckDuckBot follow robots.txt? Yes. DuckDuckGo says DuckDuckBot respects WWW::RobotRules and publishes no timing note for changes. Give DuckDuckBot its own User-agent group in robots.txt: the operator's other crawlers have their own settings. How do I know DuckDuckBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources DuckDuckGo Help Pages: Is DuckDuckBot related to DuckDuckGo? DuckDuckBot IP list (JSON, creationTime 2026-09-01, read 2026-10-24) Keep reading All AI crawlers compared DuckAssistBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## ExaSearchBot: what it does and what blocking it changes Source: https://citedby.eu/bots/exasearchbot AI crawlers / ExaSearchBot ExaSearchBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer ExaSearchBot is the crawler of Exa, a search engine built for search and retrieval, so pages can be discovered and cited. Exa says it checks robots.txt before fetching and that User-agent: ExaSearchBot with Disallow: / blocks the whole site. Every request is cryptographically signed, so verification does not depend on an IP address. Operator Exa Job Search Obeys robots.txt Yes Address list Verify tool On this page What ExaSearchBot is What the vendor says What happens if you block ExaSearchBot The robots.txt lines How to tell a real request from a fake one Compare with PerplexityBot Your own site Questions people ask Sources What ExaSearchBot is ExaSearchBot is the crawler of Exa, a search engine. Exa says its purpose is search and retrieval, so pages become discoverable and can be cited. Pages Exa fetches on a user's request rather than by crawling go out as ExaUser. Fact ExaSearchBot Operator Exa Job Search and retrieval index robots.txt token ExaSearchBot Follows robots.txt Yes. Exa says it checks robots.txt before fetching. IP addresses Verify tool Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Exa is a search engine. Similar to other search engines, Exa uses a robot (ExaSearchBot) to discover and index pages on the public web.” Source: Exa: Exa Search Crawler, read on 24 October 2026. What happens if you block ExaSearchBot Exa says User-agent: ExaSearchBot with Disallow: / blocks ExaSearchBot from the whole site. If there is no ExaSearchBot group, it follows the rules set for major search engines, otherwise the * group. The vendor says robots.txt controls crawling, not indexing, so already indexed pages are removed with a noindex meta tag or X-Robots-Tag header. The robots.txt lines To allow ExaSearchBot: User-agent: ExaSearchBot Allow: / To block ExaSearchBot: User-agent: ExaSearchBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) How to tell a real request from a fake one The vendor says every request is cryptographically signed with HTTP Message Signatures (Web Bot Auth) and gives a public key directory to verify against, and that verification holds regardless of IP address. Compare with PerplexityBot PerplexityBot : Both are search-index crawlers that feed AI answers with cited sources; ExaSearchBot signs its requests with Web Bot Auth and falls back to major search engines' rules when it has no group of its own. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). ExaSearchBot is not one of them: read your robots.txt for a ExaSearchBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does ExaSearchBot follow robots.txt? Yes. Exa says it checks robots.txt before fetching. Give ExaSearchBot its own User-agent group in robots.txt: the operator's other crawlers have their own settings. How do I know ExaSearchBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Exa: Exa Search Crawler Exa: FAQs Keep reading All AI crawlers compared PerplexityBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## ExaUser: what it does and what blocking it changes Source: https://citedby.eu/bots/exauser AI crawlers / ExaUser ExaUser: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer ExaUser is the agent Exa uses to fetch a page live when a user, or an application built on the Exa API, asks for its current content. Exa says robots.txt rules may not apply to such user-initiated fetches. Search opt-outs and automatic crawling are controlled with the ExaSearchBot token instead. Operator Exa Job User-triggered fetch Obeys robots.txt Not always Address list Verify tool On this page What ExaUser is What the vendor says What happens if you block ExaUser How to refuse ExaUser How to tell a real request from a fake one Compare with ExaSearchBot Your own site Questions people ask Sources What ExaUser is ExaUser is the agent Exa uses to fetch a page live when a user, or an application built on the Exa API, asks for its current content. Exa says it is not used to crawl the web automatically; that is ExaSearchBot. It does not try to bypass logins, paywalls or CAPTCHAs, and submits no forms. Fact ExaUser Operator Exa Job User-triggered fetch robots.txt token None documented Follows robots.txt Not always. Exa says robots.txt rules may not apply to ExaUser, because fetches are initiated by a user. IP addresses Verify tool Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Exa uses ExaUser to fetch a page live when a user, or an application built on the Exa API, asks for its current content.” Source: Exa: ExaUser, read on 24 October 2026. What happens if you block ExaUser Exa says that because fetches are initiated by a user, robots.txt rules may not apply to ExaUser. It also says ExaUser is not used to decide whether content appears in Exa search, and that search opt-outs and automatic crawling are managed with the ExaSearchBot token in robots.txt. How to refuse ExaUser Exa documents no robots.txt token for ExaUser, so there is no robots.txt line to add, and Exa says robots.txt rules may not apply to its user-initiated fetches. Exa signs every ExaUser request with Web Bot Auth, so check the signature before you trust a request that carries the name. To opt out of Exa search and automatic crawling, use the ExaSearchBot token in robots.txt (see ExaSearchBot ). The user-agent string the vendor shows: Mozilla/5.0 (compatible; ExaUser/1.0; +https://crawler.exa.ai/exauser) How to tell a real request from a fake one The vendor says every request is signed with Web Bot Auth (RFC 9421) and publishes its keys at a public directory; it does not publish an IP list. Compare with ExaSearchBot ExaSearchBot : ExaSearchBot crawls automatically to build Exa's search index and follows robots.txt, while ExaUser fetches a page only when a user or application asks and the vendor says robots.txt may not apply. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). ExaUser is not one of them, and it has no robots.txt token: look for its user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does ExaUser follow robots.txt? Not always. Exa says robots.txt rules may not apply to ExaUser, because fetches are initiated by a user. ExaUser has no documented robots.txt token, so there is no rule to add. To refuse it for certain, use a rule that checks the request's Web Bot Auth signature. How do I know ExaUser is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Exa: ExaUser Exa: Exa Search Crawler Keep reading All AI crawlers compared ExaSearchBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Google-Agent: what it does and what blocking it changes Source: https://citedby.eu/bots/google-agent AI crawlers / Google-Agent Google-Agent: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer Google-Agent is a user-triggered fetcher that Google says is used by agents hosted on Google infrastructure to navigate the web and act on a user's request. Google says user-triggered fetchers generally ignore robots.txt because a user requested the fetch. Google publishes its IP ranges in user-triggered-agents.json. Operator Google Job User-triggered fetch Obeys robots.txt Generally not Address list 20 ranges On this page What Google-Agent is What the vendor says What happens if you block Google-Agent How to refuse Google-Agent How to tell a real request from a fake one Compare with ChatGPT-User Your own site Questions people ask Sources What Google-Agent is Google-Agent is a user-triggered fetcher that Google says is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. Google lists separate mobile and desktop user agent strings. Google also says it is experimenting with the Web Bot Auth protocol using the https://agent.bot.goog identity. Fact Google-Agent Operator Google Job User-triggered fetch (agents acting for a user) robots.txt token None documented Follows robots.txt Generally not. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. IP addresses 20 ranges ( developers.google.com/static/crawling/ipranges/user-triggered-agents.json, file dated 23 October 2026) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.” Source: Google Search Central: Google user-triggered fetchers, read on 24 October 2026. What happens if you block Google-Agent Google says that because the fetch was requested by a user, user-triggered fetchers generally ignore robots.txt rules. The Google page documents no robots.txt token for Google-Agent. How to refuse Google-Agent Google documents no robots.txt token for Google-Agent, so there is no robots.txt line to add. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. To refuse it, use a firewall or CDN rule on the user agent, and check the source address against the ranges in user-triggered-agents.json. The user-agent string the vendor shows: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one The file at developers.google.com/static/crawling/ipranges/user-triggered-agents.json held 20 ranges on 24 October 2026, file dated 23 October 2026. Google says Google-Agent uses IP ranges from user-triggered-agents.json, and that user-triggered fetchers' reverse DNS mask matches ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com depending on whether the fetcher is Google or user owned. Compare with ChatGPT-User ChatGPT-User : Both act on a user's request rather than crawling on their own, and Google-Agent is described as navigating the web and performing actions for the user. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Google-Agent is not one of them, and it has no robots.txt token: look for its user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Google-Agent follow robots.txt? Generally not. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. Google-Agent has no documented robots.txt token, so there is no rule to add. To refuse it for certain, use a firewall or CDN rule that matches both the user agent and the published addresses. How do I know Google-Agent is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Google Search Central: Google user-triggered fetchers Google: user-triggered-agents.json (read 2026-10-24, creationTime 2026-10-23) Google Search Central: Overview of Google crawlers and fetchers Keep reading All AI crawlers compared ChatGPT-User AI crawler settings by platform Can ChatGPT read my website? What decides it ## Google-CloudVertexBot: what it does and what blocking it changes Source: https://citedby.eu/bots/google-cloudvertexbot AI crawlers / Google-CloudVertexBot Google-CloudVertexBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer Google-CloudVertexBot is a Google crawler that performs crawls requested by site owners to build Vertex AI Agents. Google lists it among its common crawlers, which always obey robots.txt when crawling automatically. Google says rules addressed to it have no effect on Google Search or other Google products. Operator Google Job Several jobs Obeys robots.txt Yes Address list 317 ranges On this page What Google-CloudVertexBot is What the vendor says What happens if you block Google-CloudVertexBot The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Your own site Questions people ask Sources What Google-CloudVertexBot is Google-CloudVertexBot is a Google crawler listed among its common crawlers. Google says it performs crawls requested by site owners for building Vertex AI Agents. Google shows it as a user agent substring, not a full user agent string. Fact Google-CloudVertexBot Operator Google Job Crawl requested by site owners for Vertex AI Agents robots.txt token Google-CloudVertexBot Follows robots.txt Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. IP addresses 317 ranges ( developers.google.com/static/crawling/ipranges/common-crawlers.json, file dated 23 October 2026) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents” Source: Google Search Central: Google's common crawlers, read on 24 October 2026. What happens if you block Google-CloudVertexBot Google says preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by site owners for building Vertex AI Agents. It says this has no effect on Google Search or other products. The robots.txt lines To allow Google-CloudVertexBot: User-agent: Google-CloudVertexBot Allow: / To block Google-CloudVertexBot: User-agent: Google-CloudVertexBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Google-CloudVertexBot How to tell a real request from a fake one The file at developers.google.com/static/crawling/ipranges/common-crawlers.json held 317 ranges on 24 October 2026, file dated 23 October 2026. Google says the common crawlers generally crawl from the ranges in common-crawlers.json, and that the reverse DNS name matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Compare with Googlebot Googlebot : Googlebot crawls for Google Search, while Google-CloudVertexBot only crawls when a site owner has requested a crawl for building Vertex AI Agents. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Google-CloudVertexBot is not one of them: read your robots.txt for a Google-CloudVertexBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Google-CloudVertexBot follow robots.txt? Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. How do I know Google-CloudVertexBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Google Search Central: Google's common crawlers Google: common-crawlers.json (read 2026-10-24, creationTime 2026-10-23) Google Search Central: Overview of Google crawlers and fetchers Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Google-Extended: the robots.txt token for Gemini, not a crawler Source: https://citedby.eu/bots/google-extended AI crawlers / Google-Extended Google-Extended: the robots.txt token for Gemini, not a crawler By CitedBy · Page updated 6 November 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Google-Extended is a robots.txt product token that lets publishers manage whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps. It has no HTTP user agent of its own, so it never shows in your logs. Google says it does not affect inclusion in Search or ranking. Operator Google Job Control token Obeys robots.txt Read from robots.txt only there is no separate user agent On this page What Google-Extended is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Check your own site Questions people ask Sources What Google-Extended is Most pages in this set describe a crawler you can find in a log. This one you cannot. Google states that Google-Extended "doesn't have a separate HTTP request user agent string": crawling happens with existing Google user agents, and the robots.txt token is used in a control capacity. Fact Google-Extended Operator Google Job Control token (no requests of its own) robots.txt token Google-Extended Follows robots.txt Read from robots.txt only; there is no separate user agent Published address list None: it never fetches Vendor page read 6 November 2026 What the vendor says In the vendor's words: “Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” Source: Google Search Central: Google's common crawlers (last updated 2026-10-30), read on 6 November 2026. What happens if you block it Adding Disallow: / under Google-Extended tells Google it may not use the content for training future Gemini models behind Gemini Apps and the Vertex AI API for Gemini. Google also says it does not impact inclusion in Google Search and is not a ranking signal. Google's page also names grounding, meaning content from the Google Search index given to the model at prompt time, as something this token manages. Our audit reads this token in robots.txt as one of its checks. On 30 October 2026 Google said it plans to also read a shared AI-training line, User-agent: __usage_ai-training, in 2027, and to treat an opt-out there as if the site had disallowed Google-Extended, unless the site names Google-Extended itself. Because Google-Extended also covers grounding in Gemini Apps, that opt-out would take your pages out of Gemini's grounded answers too. Naming Google-Extended with Allow: / keeps them eligible, and it also lets Google train Gemini on them: Google's page lists no separate switch for grounding. The robots.txt lines To allow it: User-agent: Google-Extended Allow: / To block it: User-agent: Google-Extended Disallow: / How to tell a real request from a fake one There is nothing to verify at the network layer: no user agent, no address list. Check the robots.txt itself, on every host, and remember that a log search for Google-Extended finds nothing whether you block it or not. Google's own example group shows the token taking an allow path and a disallow path like any other. Compare with Googlebot Googlebot : The crawler that actually fetches your pages for Search. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Why does Google-Extended never appear in my logs? Google says it has no separate HTTP request user agent string. It is a robots.txt token only, and the crawling is done by existing Google user agents. Does blocking Google-Extended hurt my Google rankings? Google says it does not impact a site's inclusion in Google Search and is not used as a ranking signal. Sources Google Search Central: Google's common crawlers (last updated 2026-10-30) Google Search Central Blog: Supporting a shared AI training control for websites Keep reading All AI crawlers compared Googlebot Is GPTBot blocked on your site? Can ChatGPT read my website? ## Google-GeminiNotebook: what it does and what blocking it changes Source: https://citedby.eu/bots/google-gemininotebook AI crawlers / Google-GeminiNotebook Google-GeminiNotebook: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Google-GeminiNotebook is the user agent of the Gemini Notebook fetcher. It requests individual URLs that users have added as sources for their projects. Google lists it among user-triggered fetchers, which generally ignore robots.txt because a user requested the fetch. Google's page names Google-NotebookLM as the former agent. Operator Google Job User-triggered fetch Obeys robots.txt Generally not Address list Reverse DNS On this page What Google-GeminiNotebook is What the vendor says What happens if you block Google-GeminiNotebook How to refuse Google-GeminiNotebook How to tell a real request from a fake one Compare with ChatGPT-User Your own site Questions people ask Sources What Google-GeminiNotebook is Google-GeminiNotebook is the user agent of the Gemini Notebook fetcher, which Google lists among its user-triggered fetchers. It requests individual URLs that users have provided as sources for their projects. Google's page lists Google-NotebookLM as the former agent, supported until August 2026. Fact Google-GeminiNotebook Operator Google Job User-triggered fetch robots.txt token None documented Follows robots.txt Generally not. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. IP addresses Reverse DNS check Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “The Gemini Notebook fetcher requests individual URLs that Gemini Notebook users have provided as sources for their projects.” Source: Google Search Central: Google user-triggered fetchers, read on 24 October 2026. What happens if you block Google-GeminiNotebook Google says that because the fetch was requested by a user, user-triggered fetchers generally ignore robots.txt rules. The Google page documents no robots.txt token for this fetcher. How to refuse Google-GeminiNotebook Google documents no robots.txt token for Google-GeminiNotebook, so there is no robots.txt line to add. To refuse it, use a firewall or CDN rule on the user agent, and check the source address against the reverse DNS check below. Google lists two user-agent strings for it, a mobile one and a desktop one, and both contain "compatible; Google-GeminiNotebook". Until August 2026 it was called Google-NotebookLM, so search your logs for both names. The user-agent string the vendor shows: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook) How to tell a real request from a fake one Google says user-triggered fetchers publish IP ranges in user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json, and that the reverse DNS mask matches ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com depending on whether the fetcher is Google or user owned. Compare with ChatGPT-User ChatGPT-User : Both fetch individual URLs because a user asked an assistant product to use them, and both vendors say these fetches may not follow robots.txt rules. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Google-GeminiNotebook is not one of them, and it has no robots.txt token: look for its user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Google-GeminiNotebook follow robots.txt? Generally not. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. Google-GeminiNotebook has no documented robots.txt token, so there is no rule to add. To refuse it for certain, use a firewall or CDN rule that matches both the user agent and the published addresses. How do I know Google-GeminiNotebook is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Google Search Central: Google user-triggered fetchers Google Search Central: Overview of Google crawlers and fetchers Keep reading All AI crawlers compared ChatGPT-User AI crawler settings by platform Can ChatGPT read my website? What decides it ## Google-Pinpoint: what it does and what blocking it changes Source: https://citedby.eu/bots/google-pinpoint AI crawlers / Google-Pinpoint Google-Pinpoint: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Google-Pinpoint is a user-triggered fetcher listed by Google. It requests individual URLs that Pinpoint users specified as sources for their personal collections of documents. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch, and documents no robots.txt token for it. Operator Google Job User-triggered fetch Obeys robots.txt Generally not Address list Reverse DNS On this page What Google-Pinpoint is What the vendor says What happens if you block Google-Pinpoint How to refuse Google-Pinpoint How to tell a real request from a fake one Compare with Google-GeminiNotebook Your own site Questions people ask Sources What Google-Pinpoint is Google-Pinpoint is a user-triggered fetcher listed by Google. It requests individual URLs that Pinpoint users specified as sources for their personal collections of documents. Google lists it with the user agent Google-Pinpoint. Fact Google-Pinpoint Operator Google Job User-triggered fetch robots.txt token None documented Follows robots.txt Generally not. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. IP addresses Reverse DNS check Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “The Google-Pinpoint fetcher requests individual URLs that Pinpoint users specified as sources for their personal collections of documents.” Source: Google Search Central: Google user-triggered fetchers, read on 24 October 2026. What happens if you block Google-Pinpoint A Pinpoint user asks for each fetch, which is why Google treats it as a user request and generally does not apply robots.txt to it. The Google page lists no robots.txt token for Google-Pinpoint. How to refuse Google-Pinpoint Google documents no robots.txt token for Google-Pinpoint, so there is no robots.txt line to add. To refuse it, use a firewall or CDN rule on the user agent, and check the source address against the reverse DNS check below. The user agent is the bare token Google-Pinpoint, with no browser string in front of it, so a plain search of your server log for that word finds every request. The user-agent string the vendor shows: Google-Pinpoint How to tell a real request from a fake one Google says user-triggered fetchers publish IP ranges in user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json, and that the reverse DNS mask matches ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com depending on whether the fetcher is Google or user owned. Compare with Google-GeminiNotebook Google-GeminiNotebook : Both fetch URLs that users supplied as sources in a Google product, and both fall under Google's user-triggered fetcher rules. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Google-Pinpoint is not one of them, and it has no robots.txt token: look for its user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Google-Pinpoint follow robots.txt? Generally not. Google says user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch. Google-Pinpoint has no documented robots.txt token, so there is no rule to add. To refuse it for certain, use a firewall or CDN rule that matches both the user agent and the published addresses. How do I know Google-Pinpoint is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Google Search Central: Google user-triggered fetchers Google Search Central: Overview of Google crawlers and fetchers Keep reading All AI crawlers compared Google-GeminiNotebook AI crawler settings by platform Can ChatGPT read my website? What decides it ## Googlebot: what it covers, its IP ranges and what a block removes Source: https://citedby.eu/bots/googlebot AI crawlers / Googlebot Googlebot: what it covers, its IP ranges and what a block removes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Googlebot is Google's main crawler for Search. Google says its common crawlers always obey robots.txt rules when crawling automatically. Crawling preferences for the Googlebot token affect Google Search, including Discover, and products such as Images, Video and News. Google publishes 317 IP ranges for it. Operator Google Job Search Obeys robots.txt Yes Address list 317 ranges On this page What Googlebot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with Google-Extended Check your own site Questions people ask Sources What Googlebot is Googlebot is the crawler every Search check starts from. Google's page lists several crawlers, and the single token Googlebot governs Search, Discover and the products that draw on the same crawl, such as Images, Video mode and News. Fact Googlebot Operator Google Job Search (also Images, Video, News, Discover) robots.txt token Googlebot Follows robots.txt Yes, always when crawling automatically Published address list 317 ranges ( developers.google.com/static/crawling/ipranges/common-crawlers.json, file dated 2026-10-23) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis. They always obey robots.txt rules when crawling automatically.” Source: Google Search Central: Google's common crawlers (last updated 2026-10-15), read on 21 October 2026. What happens if you block it A Disallow for Googlebot is not a narrow AI opt-out; it affects Google Search itself, including Discover and the features built on it, per Google's list of affected products. The narrower control for Gemini training is the separate token Google-Extended. The robots.txt lines To allow it: User-agent: Googlebot Allow: / To block it: User-agent: Googlebot Disallow: / # affects Google Search, not only AI features The user-agent string the vendor shows: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) How to tell a real request from a fake one Google publishes the ranges its common crawlers use. The Googlebot address file (googlebot.json, which now redirects to common-crawlers.json) held 317 prefixes when re-read on 25 October 2026, creation time 2026-10-23 14:45 UTC. Google also documents a reverse DNS mask for confirming a hostname, so a request is checked two ways: the address against the file, and the reverse lookup against the mask. The page itself was last updated 2026-10-15. Compare with Google-Extended Google-Extended : A control token for Gemini training; it never fetches a page. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does Googlebot obey robots.txt? Yes. Google says its common crawlers always obey robots.txt rules when crawling automatically. Will blocking Googlebot stop Gemini training? Googlebot controls Search. The Gemini training control is the separate Google-Extended token, and blocking Googlebot would also take you out of Google Search. Sources Google Search Central: Google's common crawlers (last updated 2026-10-15) developers.google.com/static/crawling/ipranges/common-crawlers.json, re-read 25 October 2026 Keep reading All AI crawlers compared Google-Extended Is GPTBot blocked on your site? Can ChatGPT read my website? ## GoogleOther: what it does and what blocking it changes Source: https://citedby.eu/bots/googleother AI crawlers / GoogleOther GoogleOther: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer GoogleOther is Google's generic crawler. Various Google product teams use it to fetch publicly accessible content, for example in one-off crawls for internal research and development. Google says rules addressed to GoogleOther do not affect any specific product. A GoogleOther group also applies to GoogleOther-Image and GoogleOther-Video unless they get their own group. Operator Google Job Several jobs Obeys robots.txt Yes Address list 317 ranges On this page What GoogleOther is What the vendor says What happens if you block GoogleOther The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Your own site Questions people ask Sources What GoogleOther is GoogleOther is Google's generic crawler, used by various Google product teams to fetch publicly accessible content, for example in one-off crawls for internal research and development. Google lists it among its common crawlers, next to Googlebot, and also documents GoogleOther-Image and GoogleOther-Video as versions optimized for image and video URLs. The Google page does not say it is used for AI training. Fact GoogleOther Operator Google Job Generic crawl for various Google product teams robots.txt token GoogleOther Follows robots.txt Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. IP addresses 317 ranges ( developers.google.com/static/crawling/ipranges/common-crawlers.json, file dated 23 October 2026) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.” Source: Google Search Central: Google's common crawlers, read on 24 October 2026. What happens if you block GoogleOther Google says crawling preferences addressed to the GoogleOther user agent do not affect any specific product. GoogleOther-Image and GoogleOther-Video list GoogleOther as a token too, and Google says you need to match only one token for a rule to apply, so a GoogleOther group also applies to them unless you give them their own group. The Googlebot token is a separate token again. The robots.txt lines To allow GoogleOther: User-agent: GoogleOther Allow: / To block GoogleOther: User-agent: GoogleOther Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A GoogleOther group also applies to GoogleOther-Image and GoogleOther-Video unless you give them their own group. The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one The file at developers.google.com/static/crawling/ipranges/common-crawlers.json held 317 ranges on 24 October 2026, file dated 23 October 2026. Google says the common crawlers generally crawl from the ranges in common-crawlers.json, and that the reverse DNS name matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Compare with Googlebot Googlebot : Googlebot crawls for Google Search and related products, while GoogleOther is a generic crawler whose robots.txt rules do not affect any specific Google product. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). GoogleOther is not one of them: read your robots.txt for a GoogleOther group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does GoogleOther follow robots.txt? Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. A GoogleOther group also applies to GoogleOther-Image and GoogleOther-Video unless you give them their own group. How do I know GoogleOther is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Google Search Central: Google's common crawlers Google: common-crawlers.json (read 2026-10-24, creationTime 2026-10-23) Google Search Central: Overview of Google crawlers and fetchers Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? What decides it ## GPTBot: what it does and what blocking it changes Source: https://citedby.eu/bots/gptbot AI crawlers / GPTBot GPTBot: what it does and what blocking it changes By CitedBy · Page updated 6 November 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer GPTBot is OpenAI's training crawler. Disallowing it in robots.txt tells OpenAI the site's content should not be used to train its generative AI foundation models. It does not control ChatGPT search, which uses OAI-SearchBot. OpenAI publishes 18 IP ranges for GPTBot. Operator OpenAI Job Training Obeys robots.txt Yes Address list 18 ranges On this page What GPTBot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with OAI-SearchBot Check your own site Questions people ask Sources What GPTBot is GPTBot is the crawler people usually mean when they say "block OpenAI". OpenAI's page says it is used to crawl content that may be used in training its models. It is one of three crawlers named in the OpenAI documentation for the ChatGPT product family, next to OAI-SearchBot and ChatGPT-User. Fact GPTBot Operator OpenAI Job Training robots.txt token GPTBot Follows robots.txt Yes, it follows robots.txt Published address list 18 ranges ( openai.com/gptbot.json, file dated 2026-09-22) Vendor page read 6 November 2026 What the vendor says In the vendor's words: “GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.” Source: OpenAI: Overview of OpenAI Crawlers, read on 6 November 2026. What happens if you block it OpenAI frames a GPTBot Disallow as a signal about training: the content should not be used in training generative AI foundation models. The same page gives an example of a site allowing OAI-SearchBot to appear in search results while disallowing GPTBot, so a block here does not by itself take you out of ChatGPT's search answers. Whether to block is a business choice; both are legitimate. On 30 October 2026 OpenAI said it will also honour a shared AI-training line, User-agent: __usage_ai-training, and aims to do so "in the coming months". It "will not replace" the controls OpenAI already provides through robots.txt, so a GPTBot rule keeps working. The robots.txt lines To allow it: User-agent: GPTBot Allow: / To block it: User-agent: GPTBot Disallow: / The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot How to tell a real request from a fake one The GPTBot list at openai.com/gptbot.json is the shortest of the three OpenAI files we counted: 18 prefixes on 21 October 2026, creation time 2026-09-22. When a request names GPTBot but comes from an address outside that list, the name alone proves nothing. Compare with OAI-SearchBot OAI-SearchBot : Allow it and disallow GPTBot to stay in ChatGPT search while opting out of training. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask If I block GPTBot, will ChatGPT stop mentioning my shop? Not through GPTBot alone. OpenAI treats its crawler settings as independent and says search appearance is managed through OAI-SearchBot. Check both tokens in your robots.txt. How do I know GPTBot is really blocked? Robots.txt is only one layer. A CDN or firewall rule can also turn the crawler away. Our guide to the four places GPTBot can be stopped shows a two-request test for each. Sources OpenAI: Overview of OpenAI Crawlers openai.com/gptbot.json, read 21 October 2026 OpenAI: Making it easier for websites to express AI training preferences Keep reading All AI crawlers compared OAI-SearchBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## ICC-Crawler: what it does and what blocking it changes Source: https://citedby.eu/bots/icc-crawler AI crawlers / ICC-Crawler ICC-Crawler: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer ICC-Crawler is run by the Universal Communication Research Institute at Japan's NICT. NICT says it uses collected pages for research on information processing, including multilingual translation, information analysis and artificial intelligence. It supports the Robots Exclusion Protocol: User-agent: ICC-Crawler with Disallow: / could disable collection from all pages. Operator NICT (Universal Communication Research Institute) Job Training Obeys robots.txt Yes Address list Listed on its page On this page What ICC-Crawler is What the vendor says What happens if you block ICC-Crawler The robots.txt lines How to tell a real request from a fake one Compare with GPTBot Your own site Questions people ask Sources What ICC-Crawler is ICC-Crawler is a crawler run by the Universal Communication Research Institute at Japan's National Institute of Information and Communications Technology (NICT). NICT says it uses collected pages for research and development of information processing technologies, including multilingual translation, information analysis and artificial intelligence. NICT may also provide collected information or research products to third parties, to the extent permitted by law. Fact ICC-Crawler Operator NICT (Universal Communication Research Institute) Job Research data collection (includes AI research and development) robots.txt token ICC-Crawler Follows robots.txt Yes, and with a Crawl-Delay it uses the larger of that interval and its own minimum. IP addresses Listed on the vendor's page ( ucri.nict.go.jp/en/icccrawler.html ) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “NICT uses the information it collects for research and development of advanced information processing technologies including multilingual translation, information analysis, and artificial intelligence technologies.” Source: NICT Universal Communication Research Institute: ICC-Crawler, read on 24 October 2026. What happens if you block ICC-Crawler NICT says ICC-Crawler supports the Robots Exclusion Protocol and that adding User-agent: ICC-Crawler with Disallow: / to robots.txt could disable collection from all pages. It also lets site owners block a path or file type, or allow selected paths under a full Disallow. If access continues after that, NICT asks site owners to contact it so it can stop collecting from the host. The robots.txt lines To allow ICC-Crawler: User-agent: ICC-Crawler Allow: / To block ICC-Crawler: User-agent: ICC-Crawler Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The vendor's page shows no full user-agent string for ICC-Crawler. How to tell a real request from a fake one The vendor page lists the crawler's IP addresses in plain text; it does not publish a machine-readable list. Compare with GPTBot GPTBot : Both collect pages that feed AI development, but ICC-Crawler is run by a Japanese public research institute for its own research and publishes two crawler IP addresses on its page, while GPTBot is run by a commercial company. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). ICC-Crawler is not one of them: read your robots.txt for a ICC-Crawler group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does ICC-Crawler follow robots.txt? Yes, and with a Crawl-Delay it uses the larger of that interval and its own minimum. How do I know ICC-Crawler is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources NICT Universal Communication Research Institute: ICC-Crawler Keep reading All AI crawlers compared GPTBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## ImagesiftBot: what it does and what blocking it changes Source: https://citedby.eu/bots/imagesiftbot AI crawlers / ImagesiftBot ImagesiftBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer ImagesiftBot is the crawler of ImageSift, a Hive product. It collects publicly available images with the host URL, page text and image alt text for an index used to search for similar images. The vendor says it respects robots.txt and Crawl-delay, and follows the Googlebot rules when none target it. Operator Hive (ImageSift) Job Search Obeys robots.txt Yes Address list None On this page What ImagesiftBot is What the vendor says What happens if you block ImagesiftBot The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Your own site Questions people ask Sources What ImagesiftBot is ImageSiftBot is the crawler of ImageSift, a Hive product. It collects publicly available images, along with the host URL, page text and image alt text. ImageSift stores the data in an index that its products use for search and retrieval of similar images. Fact ImagesiftBot Operator Hive (ImageSift) Job Image index for similar-image search robots.txt token ImagesiftBot Follows robots.txt Yes, and it supports Crawl-delay. IP addresses None published Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products” Source: Imagesift by Hive: ImageSift Bot, read on 24 October 2026. What happens if you block ImagesiftBot The vendor says standard robots.txt directives that target ImagesiftBot are respected, and that it supports Crawl-delay. If no rule targets ImagesiftBot but a rule targets Googlebot, it follows the Googlebot rules. The page also gives support@imagesift.com for opt-out requests. The robots.txt lines To allow ImagesiftBot: User-agent: ImagesiftBot Allow: / To block ImagesiftBot: User-agent: ImagesiftBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) How to tell a real request from a fake one The vendor page does not publish an IP list or a verification method. Compare with Googlebot Googlebot : ImagesiftBot falls back to Googlebot's robots.txt rules when it has none of its own, and it collects images for a similar-image index rather than for a general web search. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). ImagesiftBot is not one of them: read your robots.txt for a ImagesiftBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does ImagesiftBot follow robots.txt? Yes, and it supports Crawl-delay. How do I know ImagesiftBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Imagesift by Hive: ImageSift Bot Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Kagibot: what it does and what blocking it changes Source: https://citedby.eu/bots/kagibot AI crawlers / Kagibot Kagibot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Kagibot is the web crawler for the Kagi search engine. Kagi says standard robots.txt directives that target Kagibot are respected, and that when no rule targets Kagibot but one targets Googlebot, Kagibot follows the Googlebot directives. Kagi publishes its user-agent string and the IP addresses its requests come from. Operator Kagi Job Search Obeys robots.txt Yes Address list Listed on its page On this page What Kagibot is What the vendor says What happens if you block Kagibot The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Your own site Questions people ask Sources What Kagibot is Kagibot is the web crawler for the Kagi search engine. Kagi publishes its user-agent string and the IP addresses that Kagibot requests originate from. Fact Kagibot Operator Kagi Job Search robots.txt token Kagibot Follows robots.txt Yes, standard robots.txt directives targeting Kagibot are respected. IP addresses Listed on the vendor's page ( kagi.com/bot ) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Standard directives in robots.txt that target Kagibot are respected.” Source: Kagi: About KagiBot, read on 24 October 2026. What happens if you block Kagibot Kagi says standard robots.txt directives that target Kagibot are respected, for example Disallow rules for a path. If there is no rule for Kagibot but there is one for Googlebot, Kagibot follows the Googlebot directives. The robots.txt lines To allow Kagibot: User-agent: Kagibot Allow: / To block Kagibot: User-agent: Kagibot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot) How to tell a real request from a fake one Kagi lists the IP addresses (with kagibot.org hostnames) that requests from Kagibot originate from on its bot page. Compare with Googlebot Googlebot : Kagibot follows its own robots.txt rules, and falls back to the Googlebot rules when none target Kagibot. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Kagibot is not one of them: read your robots.txt for a Kagibot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Kagibot follow robots.txt? Yes, standard robots.txt directives targeting Kagibot are respected. How do I know Kagibot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Kagi: About KagiBot (read 2026-10-24) Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? What decides it ## LinkupBot: what it does and what blocking it changes Source: https://citedby.eu/bots/linkupbot AI crawlers / LinkupBot LinkupBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer LinkupBot is the web crawler of Linkup. It fetches publicly accessible pages to build the Linkup search index, which Linkup says powers search and web grounding for AI applications. Linkup says User-agent: LinkupBot with Disallow: / blocks the whole site, and that robots.txt is not an access-control mechanism. Operator Linkup Job Search Obeys robots.txt Yes Address list Listed on its page On this page What LinkupBot is What the vendor says What happens if you block LinkupBot The robots.txt lines How to tell a real request from a fake one Compare with PerplexityBot Your own site Questions people ask Sources What LinkupBot is LinkupBot is the web crawler of Linkup. It fetches publicly accessible pages to build and maintain the Linkup search index, which Linkup says powers search and web grounding for AI applications. Fact LinkupBot Operator Linkup Job Search index for AI web search robots.txt token LinkupBot Follows robots.txt Yes. Linkup says it respects applicable robots.txt rules. IP addresses Listed on the vendor's page ( www.linkup.so/linkupbot-ips.txt ) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “LinkupBot is Linkup’s web crawler. It discovers and retrieves publicly accessible webpages to build and maintain the Linkup web search index, which powers search and web grounding for AI applications.” Source: Linkup: LinkupBot, read on 24 October 2026. What happens if you block LinkupBot Adding User-agent: LinkupBot with Disallow: / to robots.txt blocks the whole site, according to the vendor. The vendor also says robots rules apply per host, that allowing LinkupBot through a firewall does not override robots.txt, and that robots.txt is not an access-control mechanism. The robots.txt lines To allow LinkupBot: User-agent: LinkupBot Allow: / To block LinkupBot: User-agent: LinkupBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: LinkupBot/1.0 (LinkupBot for web indexing; https://www.linkup.so/bot; bot@linkup.so) How to tell a real request from a fake one The vendor says requests come from the CIDR ranges in its published list and to verify with both a source IP in that list and a user-agent containing the LinkupBot token. Compare with PerplexityBot PerplexityBot : Both build a search index that feeds AI answers; LinkupBot's index serves AI applications through Linkup's search and grounding service, and its IP list is a plain text file rather than JSON. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). LinkupBot is not one of them: read your robots.txt for a LinkupBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does LinkupBot follow robots.txt? Yes. Linkup says it respects applicable robots.txt rules. How do I know LinkupBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Linkup: LinkupBot Linkup: linkupbot-ips.txt (plain text, read 2026-10-24) Keep reading All AI crawlers compared PerplexityBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Meta-ExternalAgent: Meta's AI crawler and its four siblings Source: https://citedby.eu/bots/meta-externalagent AI crawlers / Meta-ExternalAgent Meta-ExternalAgent: Meta's AI crawler and its four siblings By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Meta-ExternalAgent is Meta's crawler for use cases such as training foundation AI models or improving products by indexing content directly. Meta's page, updated 21 May 2026, lists five agents. It says Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt, and that changes can take up to 24 hours. Operator Meta Job AI model training Obeys robots.txt Yes Address list None read On this page What Meta-ExternalAgent is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with GPTBot Check your own site Questions people ask Sources What Meta-ExternalAgent is Meta's page lists five crawlers. FacebookExternalHit fetches a page when someone shares a link on Facebook, Instagram or Messenger, to cache its title, description and thumbnail. Meta-WebIndexer improves Meta AI search results, and Meta says allowing it helps Meta cite and link to your content in Meta AI's responses. Meta-ExternalAds crawls for advertising and business products. Meta-ExternalAgent is the one for training foundation AI models or indexing. Meta-ExternalFetcher fetches individual links at a user's request, including helping AI navigate websites to complete tasks for users. Fact Meta-ExternalAgent Operator Meta Job AI model training and content indexing robots.txt token meta-externalagent Follows robots.txt Yes Published address list None read Vendor page read 21 October 2026 What the vendor says In the vendor's words: “The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Source: Meta for Developers: Meta Web Crawlers, read on 21 October 2026. What happens if you block it Disallowing meta-externalagent affects that agent only: Meta's example gives a separate group to each of two tokens, and the other four have their own. The costly one to block by mistake is FacebookExternalHit, because it builds the link preview when someone shares your page. Two of the five may ignore your robots.txt: Meta says Meta-ExternalFetcher may bypass robots.txt because it performs fetches requested by the user, and FacebookExternalHit may bypass it for security or integrity checks such as malware checks. The robots.txt lines To allow it: User-agent: meta-webindexer Allow: / User-agent: meta-externalagent Allow: / To block it: User-agent: meta-externalagent Disallow: / The user-agent string the vendor shows: meta-externalagent/1.1 How to tell a real request from a fake one Meta's page gives the user-agent strings (meta-externalagent/1.1 and the version with a link) and does not publish an IP list in the text we read, so we state no count. For FacebookExternalHit Meta says to allow-list either the user-agent strings or the IP addresses, and calls the addresses the more secure option. Meta says crawlers may cache robots.txt for up to 24 hours, so allow that long before judging a change. Questions go to webmasters@meta.com. Compare with GPTBot GPTBot : OpenAI's training crawler, which honours robots.txt without Meta's bypass cases. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Which Meta crawler trains AI models? Meta describes Meta-ExternalAgent as crawling for use cases such as training foundation AI models or improving products by indexing content directly. Does Meta-ExternalFetcher obey robots.txt? Not always. Meta says it may bypass robots.txt rules because it fetches links at a user's request. How long does a robots.txt change take? Meta says to allow up to 24 hours, because crawlers may cache the file for up to 24 hours. Sources Meta for Developers: Meta Web Crawlers, read 21 October 2026 Keep reading All AI crawlers compared GPTBot AI crawler settings by platform Can ChatGPT read my website? ## Meta-ExternalFetcher: what it does and what blocking it changes Source: https://citedby.eu/bots/meta-externalfetcher AI crawlers / Meta-ExternalFetcher Meta-ExternalFetcher: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Meta-ExternalFetcher is a Meta crawler that fetches individual links at a user's request. Meta says it supports product functions such as helping AI navigate websites to complete tasks for users, and that it may bypass robots.txt because the fetches are requested by a user. Robots.txt changes can take up to 24 hours. Operator Meta Job User-triggered fetch Obeys robots.txt Not always Address list None On this page What Meta-ExternalFetcher is What the vendor says What happens if you block Meta-ExternalFetcher The robots.txt lines How to tell a real request from a fake one Compare with Meta-ExternalAgent Your own site Questions people ask Sources What Meta-ExternalFetcher is Meta-ExternalFetcher is a Meta crawler that fetches individual links at a user's request. Meta says it supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users. Meta documents it alongside Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalAds and FacebookExternalHit on one page. Fact Meta-ExternalFetcher Operator Meta Job User-triggered fetch robots.txt token Meta-ExternalFetcher Follows robots.txt Not always. Meta says Meta-ExternalFetcher may bypass robots.txt because fetches are requested by a user. IP addresses None published Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “The Meta-ExternalFetcher crawler fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities—including helping AI navigate websites to complete tasks for users. Accordingly, this crawler may bypass robots.txt rules.” Source: Meta: Meta Web Crawlers, read on 24 October 2026. What happens if you block Meta-ExternalFetcher Meta says you can block its crawlers by adding a disallow for the relevant crawler to robots.txt. Meta also says Meta-ExternalFetcher may bypass robots.txt because it performs fetches that were requested by the user. Meta says changes to robots.txt can take up to 24 hours to take effect because crawlers may cache the file for up to 24 hours. The robots.txt lines To allow Meta-ExternalFetcher: User-agent: Meta-ExternalFetcher Allow: / To block Meta-ExternalFetcher: User-agent: Meta-ExternalFetcher Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: meta-externalfetcher/1.1 How to tell a real request from a fake one Meta's crawler page does not list IP addresses or a verification method for this crawler; it identifies the crawler by its user-agent string. Compare with Meta-ExternalAgent Meta-ExternalAgent : Meta-ExternalAgent crawls the web in the background, including for training foundation AI models, while Meta-ExternalFetcher fetches single links a user asked for and may bypass robots.txt. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Meta-ExternalFetcher is not one of them: read your robots.txt for a Meta-ExternalFetcher group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Meta-ExternalFetcher follow robots.txt? Not always. Meta says Meta-ExternalFetcher may bypass robots.txt because fetches are requested by a user. A rule for Meta-ExternalFetcher in robots.txt may therefore not stop it. To refuse it for certain, use a firewall or CDN rule on the user agent; Meta publishes no address list to check it against. How do I know Meta-ExternalFetcher is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Meta: Meta Web Crawlers Keep reading All AI crawlers compared Meta-ExternalAgent AI crawler settings by platform Can ChatGPT read my website? What decides it ## Meta-WebIndexer: what it does and what blocking it changes Source: https://citedby.eu/bots/meta-webindexer AI crawlers / Meta-WebIndexer Meta-WebIndexer: what it does and what blocking it changes By CitedBy · Page updated 24 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Meta-WebIndexer is a Meta crawler that navigates the web to improve Meta AI search result quality. Meta says allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses. Meta says you block its crawlers with a disallow for the relevant crawler, and a change can take up to 24 hours. Operator Meta Job Search Obeys robots.txt Yes Address list None On this page What Meta-WebIndexer is What the vendor says What happens if you block Meta-WebIndexer The robots.txt lines How to tell a real request from a fake one Compare with Meta-ExternalAgent Your own site Questions people ask Sources What Meta-WebIndexer is Meta-WebIndexer is a Meta crawler that navigates the web to improve Meta AI search result quality for users. Meta says allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses. Meta documents it alongside Meta-ExternalAgent, Meta-ExternalFetcher, Meta-ExternalAds and FacebookExternalHit on one page. Fact Meta-WebIndexer Operator Meta Job Search robots.txt token Meta-WebIndexer Follows robots.txt Yes, changes may take up to 24 hours. IP addresses None published Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI.” Source: Meta: Meta Web Crawlers, read on 24 October 2026. What happens if you block Meta-WebIndexer Meta says you can block its crawlers by adding a disallow for the relevant crawler to robots.txt, and that robots.txt changes can take up to 24 hours to take effect. Meta says allowing Meta-WebIndexer helps it cite and link to your content in Meta AI's responses. Meta's page does not say whether a disallow for Meta-WebIndexer affects its other crawlers. The robots.txt lines To allow Meta-WebIndexer: User-agent: Meta-WebIndexer Allow: / To block Meta-WebIndexer: User-agent: Meta-WebIndexer Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: meta-webindexer/1.1 How to tell a real request from a fake one Meta's crawler page does not list IP addresses or a verification method for this crawler; it identifies the crawler by its user-agent string. Compare with Meta-ExternalAgent Meta-ExternalAgent : Meta-WebIndexer is for Meta AI search results and citations, while Meta-ExternalAgent is described by Meta as crawling for uses such as training foundation AI models or indexing content directly. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Meta-WebIndexer is not one of them: read your robots.txt for a Meta-WebIndexer group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Meta-WebIndexer follow robots.txt? Yes, changes may take up to 24 hours. Give Meta-WebIndexer its own User-agent group in robots.txt: the operator's other crawlers have their own settings. How do I know Meta-WebIndexer is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Meta: Meta Web Crawlers Keep reading All AI crawlers compared Meta-ExternalAgent AI crawler settings by platform Can ChatGPT read my website? What decides it ## MistralAI-User, MistralAI-Index and MistralAI-Training explained Source: https://citedby.eu/bots/mistralai-user AI crawlers / MistralAI-User MistralAI-User, MistralAI-Index and MistralAI-Training explained By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer MistralAI-User is the Mistral agent that may visit a page when a user asks Vibe a question and link the source in its answer. Mistral says it is not used to crawl automatically or for generative AI training. Mistral's page also lists MistralAI-Index and MistralAI-Training, each with its own token. Operator Mistral Job User-triggered fetch Obeys robots.txt Opt-out through robots.txt Address list 4 ranges On this page What MistralAI-User is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with ChatGPT-User Check your own site Questions people ask Sources What MistralAI-User is Mistral's crawler page lists three agents: MistralAI-User for user actions in Vibe, MistralAI-Index for indexing content for Mistral search, and MistralAI-Training, which crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow each in robots.txt. Fact MistralAI-User Operator Mistral Job User-triggered fetch robots.txt token MistralAI-User Follows robots.txt Yes. Mistral says you can opt out through robots.txt. Published address list 4 ranges ( mistral.ai/mistralai-user-ips.json, file dated 2025-02-19) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “MistralAI-User is for user actions in Vibe. When users ask Vibe a question, it may visit a web page to help answer and include a link to the source in its response.” Source: Mistral Docs: Mistral crawlers, read on 21 October 2026. What happens if you block it Disallowing MistralAI-User stops that agent opening your page for a Vibe user's question; it leaves the index and training agents alone, which have their own tokens. Mistral says the user agent is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training, and that content crawled by the index agent is not used for generative AI training of any kind. The robots.txt lines To allow it: User-agent: MistralAI-User Allow: / To block it: User-agent: MistralAI-User Disallow: / User-agent: MistralAI-Training Disallow: / The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) How to tell a real request from a fake one Mistral publishes separate address files for the agents. mistral.ai/mistralai-user-ips.json held 4 prefixes (creation time 2025-02-19) and the index file 2 prefixes (creation time 2026-04-19) on 21 October 2026. Two entries for an index crawler is a list a firewall rule can hold exactly. Compare with ChatGPT-User ChatGPT-User : The same job at OpenAI, with a different robots.txt promise. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Which Mistral agent trains models? Mistral's page says MistralAI-Training crawls web content to help build datasets for training its generative AI models. MistralAI-User and MistralAI-Index are described as not used for training. Is Vibe a crawler? Vibe is Mistral's product. MistralAI-User is the agent that may visit a page on behalf of a Vibe user. Sources Mistral Docs: Mistral crawlers mistral.ai/mistralai-user-ips.json, read 21 October 2026 Keep reading All AI crawlers compared ChatGPT-User Is GPTBot blocked on your site? Can ChatGPT read my website? ## MojeekBot: what it does and what blocking it changes Source: https://citedby.eu/bots/mojeekbot AI crawlers / MojeekBot MojeekBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer MojeekBot is the web crawler for the Mojeek search engine. Mojeek says it obeys the Robots Exclusion Standard, using the first record whose User-Agent contains MojeekBot, and does not support the nonstandard crawl-delay directive. It should not request more than one page from a site in the same second. Operator Mojeek Job Search Obeys robots.txt Yes Address list 1 ranges On this page What MojeekBot is What the vendor says What happens if you block MojeekBot The robots.txt lines How to tell a real request from a fake one Compare with Bingbot Your own site Questions people ask Sources What MojeekBot is MojeekBot is the web crawler for the Mojeek search engine. Mojeek says it should not request more than one page from a site within the same 1 second period. It obeys the noindex, nocache and nofollow meta tags. Fact MojeekBot Operator Mojeek Job Search robots.txt token MojeekBot Follows robots.txt Yes. Mojeek says MojeekBot obeys the Robots Exclusion Standard; it does not support crawl-delay. IP addresses 1 ranges ( www.mojeek.com/mojeekbot.json, file dated 14 May 2025) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “MojeekBot is the web crawler for the Mojeek search engine.” Source: Mojeek: MojeekBot, read on 24 October 2026. What happens if you block MojeekBot Mojeek says MojeekBot obeys the first robots.txt record whose User-Agent contains MojeekBot, and otherwise the first record for User-Agent *. It will not retrieve URLs containing a disallowed string. It does not support the nonstandard crawl-delay directive. The robots.txt lines To allow MojeekBot: User-agent: MojeekBot Allow: / To block MojeekBot: User-agent: MojeekBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: MojeekBot How to tell a real request from a fake one The file at www.mojeek.com/mojeekbot.json held 1 ranges on 24 October 2026, file dated 14 May 2025. Mojeek says to reverse-resolve the visiting IP to a name in the mojeek.com domain and forward-resolve it back to the same IP, or to match the IP against its published list of MojeekBot IP addresses. Compare with Bingbot Bingbot : Both crawl for a search engine; Mojeek publishes a one-range IP list and a reverse DNS check, and does not support crawl-delay. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). MojeekBot is not one of them: read your robots.txt for a MojeekBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does MojeekBot follow robots.txt? Yes. Mojeek says MojeekBot obeys the Robots Exclusion Standard; it does not support crawl-delay. How do I know MojeekBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Mojeek: MojeekBot MojeekBot IP list (JSON, creationTime 2025-05-14, read 2026-10-24) Keep reading All AI crawlers compared Bingbot AI crawler settings by platform Can ChatGPT read my website? What decides it ## OAI-SearchBot: what it is and how to allow or block it Source: https://citedby.eu/bots/oai-searchbot AI crawlers / OAI-SearchBot OAI-SearchBot: what it is and how to allow or block it By CitedBy · Page updated 22 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer OAI-SearchBot is OpenAI's crawler for ChatGPT search. Sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. It obeys robots.txt under the token OAI-SearchBot, and OpenAI publishes 39 IP ranges for it. Operator OpenAI Job Search Obeys robots.txt Yes Address list 39 ranges On this page What OAI-SearchBot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Compare with GPTBot Check your own site Questions people ask Sources What OAI-SearchBot is OpenAI runs several crawlers and says each setting is independent of the others. OAI-SearchBot is the one that decides whether your pages can be surfaced in ChatGPT's search features. It is not the training crawler (that is GPTBot ) and it is not the fetcher that opens a page when a person asks about it (that is ChatGPT-User ). Fact OAI-SearchBot Operator OpenAI Job Search robots.txt token OAI-SearchBot Follows robots.txt Yes, it follows robots.txt Published address list 39 ranges ( openai.com/searchbot.json, file dated 2026-01-02) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” Source: OpenAI: Overview of OpenAI Crawlers, read on 21 October 2026. What happens if you block it OpenAI's own wording is that opted-out sites "will not be shown in ChatGPT search answers". That is the cost: a shop that blocks OAI-SearchBot gives up the part of ChatGPT that links to sources. OpenAI recommends the opposite for sites that want to appear: allow OAI-SearchBot in robots.txt and allow requests from its published IP ranges. The robots.txt lines To allow it: User-agent: OAI-SearchBot Allow: / To block it: User-agent: OAI-SearchBot Disallow: / The user-agent string the vendor shows: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot How to tell a real request from a fake one OpenAI publishes the addresses OAI-SearchBot uses at openai.com/searchbot.json. On 21 October 2026 that file held 39 prefixes and carried the creation time 2026-01-02. Match the address in your server log against those prefixes before you trust a request that only claims the name in its user-agent. OpenAI also says that when it fetches robots.txt it may add a robots.txt marker to the user-agent string, so those requests are easy to tell apart in logs without paths. Compare with GPTBot GPTBot : GPTBot is for training; blocking it does not remove you from ChatGPT search because the settings are independent. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does blocking GPTBot also block OAI-SearchBot? No. OpenAI states that each setting is independent of the others, so a robots.txt group for GPTBot does not change how OAI-SearchBot treats your site. Add a separate group for each token you want to control. Why would a firewall block OAI-SearchBot when robots.txt allows it? Robots.txt is only a policy file. A CDN, WAF or host rule can still refuse the request before the server answers. OpenAI asks site owners to allow its published IP ranges for this reason. Sources OpenAI: Overview of OpenAI Crawlers openai.com/searchbot.json, read 21 October 2026 Keep reading All AI crawlers compared GPTBot Is GPTBot blocked on your site? Can ChatGPT read my website? ## Perplexity-User: what it does and what blocking it changes Source: https://citedby.eu/bots/perplexity-user AI crawlers / Perplexity-User Perplexity-User: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Perplexity-User is Perplexity's user-action fetcher: when a user asks a question, it may visit a page to answer accurately and link to it. Perplexity says it is not used for web crawling or for training AI models, and that it generally ignores robots.txt rules because a user requested the fetch. Operator Perplexity Job User-triggered fetch Obeys robots.txt Generally not Address list 4 ranges On this page What Perplexity-User is What the vendor says What happens if you block Perplexity-User The robots.txt lines How to tell a real request from a fake one Compare with PerplexityBot Your own site Questions people ask Sources What Perplexity-User is Perplexity-User is Perplexity's user-action fetcher. When a user asks Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response. Perplexity says it is not used for web crawling or to collect content for training AI foundation models, and it is separate from PerplexityBot, which surfaces and links websites in Perplexity search results. Fact Perplexity-User Operator Perplexity Job User-triggered fetch robots.txt token Perplexity-User Follows robots.txt Generally not. Perplexity says this fetcher generally ignores robots.txt rules because a user requested the fetch. IP addresses 4 ranges ( www.perplexity.com/perplexity-user.json, file dated 17 October 2025) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Source: Perplexity: Perplexity Crawlers, read on 24 October 2026. What happens if you block Perplexity-User Perplexity lists Perplexity-User as a robots.txt token and says it controls which sites these user requests can access. Perplexity also says this fetcher generally ignores robots.txt rules because a user requested the fetch. Perplexity says each of its robots.txt settings works independently and changes may take up to 24 hours to take effect. The robots.txt lines To allow Perplexity-User: User-agent: Perplexity-User Allow: / To block Perplexity-User: User-agent: Perplexity-User Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers. The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) How to tell a real request from a fake one The file at www.perplexity.com/perplexity-user.json held 4 ranges on 24 October 2026, file dated 17 October 2025. Perplexity publishes the Perplexity-User IP addresses as a JSON file and says to use it as the source of truth for allow rules, combined with the user-agent string. Compare with PerplexityBot PerplexityBot : PerplexityBot surfaces and links sites in Perplexity search results, while Perplexity-User fetches a page when a user's question needs it and generally ignores robots.txt. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Perplexity-User is not one of them: read your robots.txt for a Perplexity-User group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Perplexity-User follow robots.txt? Generally not. Perplexity says this fetcher generally ignores robots.txt rules because a user requested the fetch. A rule for Perplexity-User in robots.txt may therefore not stop it. To refuse it for certain, use a firewall or CDN rule that matches both the user agent and the published addresses. How do I know Perplexity-User is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Perplexity: Perplexity Crawlers Perplexity: Perplexity-User IP addresses JSON (read 2026-10-24) Keep reading All AI crawlers compared PerplexityBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## PerplexityBot vs Perplexity-User: differences and how to allow it Source: https://citedby.eu/bots/perplexitybot AI crawlers / PerplexityBot PerplexityBot: how it differs from Perplexity-User and how to allow it By CitedBy · Page updated 30 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer PerplexityBot is Perplexity's crawler for surfacing and linking websites in its search results. Perplexity says it is not used to crawl content for AI foundation models. It follows robots.txt, changes can take up to 24 hours to apply, and Perplexity publishes 8 IP ranges for it. Operator Perplexity Job Search Obeys robots.txt Yes Address list 8 ranges On this page What PerplexityBot is What the vendor says What happens if you block it The robots.txt lines How to tell a real request from a fake one Test it on your own site Compare with Perplexity-User Check your own site Questions people ask Sources What PerplexityBot is Perplexity names two agents. PerplexityBot builds the search index; Perplexity-User visits a page when a person asks. Perplexity's page contains configuration guidelines for popular WAF providers because a firewall can stop the crawler before robots.txt matters. Fact PerplexityBot Operator Perplexity Job Search robots.txt token PerplexityBot Follows robots.txt Yes, it follows robots.txt Published address list 8 ranges ( perplexity.com/perplexitybot.json, file dated 2025-02-07) Vendor page read 21 October 2026 What the vendor says In the vendor's words: “PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Source: Perplexity: Perplexity Crawlers, read on 21 October 2026. What happens if you block it Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges if you want to appear in its search results. It also says changes to your robots.txt can take up to 24 hours to be reflected in its systems, so a check made an hour after an edit proves little. The robots.txt lines To allow it: User-agent: PerplexityBot Allow: / To block it: User-agent: PerplexityBot Disallow: / The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) How to tell a real request from a fake one Perplexity's file at perplexity.com/perplexitybot.json held 8 prefixes on 21 October 2026, creation time 2025-02-07. Its WAF section suggests a rule that combines the user-agent with the IP condition, so a request claiming the name from another address does not get through. Test it on your own site Two requests show whether a firewall treats the crawler's name differently from a normal visitor. Replace yourshop.com with your domain: curl -s -o /dev/null -w "%{http_code}\n" https://yourshop.com/ curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourshop.com/ Two 200s mean the name alone is not refused. A 403 or a challenge page on the second request means a rule keyed on the user-agent: look in your CDN, firewall or security plugin. The test comes from your own address, so it cannot show a rule that depends on Perplexity's published ranges. For that, read your server log (this line works on the usual nginx and Apache combined format): grep -i perplexitybot access.log | awk '{print $9}' | sort | uniq -c A run of 403 or 429 codes against addresses inside the 8 published ranges means the crawler is being refused even though robots.txt allows it. Perplexity says changes can take up to 24 hours to apply, so run the log check a day after you change a rule. Compare with Perplexity-User Perplexity-User : The user-triggered fetcher; Perplexity says it generally ignores robots.txt. Check your own site The free check in the box above reads robots.txt for the crawlers in this set and sends one live request per crawler, so a rule that robots.txt allows but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask How long does a robots.txt change take for PerplexityBot? Perplexity says it may take up to 24 hours for its systems to reflect a change. Does PerplexityBot train AI models? Perplexity says it is not used to crawl content for AI foundation models. Sources Perplexity: Perplexity Crawlers perplexity.com/perplexitybot.json, read 21 October 2026 Keep reading All AI crawlers compared Is GPTBot blocked on your site? Can ChatGPT read my website? ## Storebot-Google: what it does and what blocking it changes Source: https://citedby.eu/bots/storebot-google AI crawlers / Storebot-Google Storebot-Google: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer Storebot-Google is the crawler Google documents as Google StoreBot among its common crawlers. Google says a Disallow addressed to its token affects all surfaces of Google Shopping, such as the Shopping tab in Google Search and Google Shopping. Googlebot is a separate token with its own affected products. Operator Google Job Search Obeys robots.txt Yes Address list 317 ranges On this page What Storebot-Google is What the vendor says What happens if you block Storebot-Google The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Your own site Questions people ask Sources What Storebot-Google is Storebot-Google is the Google crawler documented as Google StoreBot among its common crawlers. Google says rules addressed to it affect all surfaces of Google Shopping. Google lists separate desktop and mobile user agent strings for it. Fact Storebot-Google Operator Google Job Search (Google Shopping surfaces) robots.txt token Storebot-Google Follows robots.txt Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. IP addresses 317 ranges ( developers.google.com/static/crawling/ipranges/common-crawlers.json, file dated 23 October 2026) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping” Source: Google Search Central: Google's common crawlers, read on 24 October 2026. What happens if you block Storebot-Google Google says a Disallow addressed to the Storebot-Google token affects all surfaces of Google Shopping, such as the Shopping tab in Google Search and Google Shopping. The page lists Googlebot as a separate token with its own affected products. The robots.txt lines To allow Storebot-Google: User-agent: Storebot-Google Allow: / To block Storebot-Google: User-agent: Storebot-Google Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 How to tell a real request from a fake one The file at developers.google.com/static/crawling/ipranges/common-crawlers.json held 317 ranges on 24 October 2026, file dated 23 October 2026. Google says the common crawlers generally crawl from the ranges in common-crawlers.json, and that the reverse DNS name matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Compare with Googlebot Googlebot : Googlebot's rules affect Google Search and related products, while Storebot-Google's rules affect Google Shopping surfaces. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Storebot-Google is not one of them: read your robots.txt for a Storebot-Google group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does Storebot-Google follow robots.txt? Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. How do I know Storebot-Google is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Google Search Central: Google's common crawlers Google: common-crawlers.json (read 2026-10-24, creationTime 2026-10-23) Google Search Central: Overview of Google crawlers and fetchers Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? What decides it ## YandexBot: what it does and what blocking it changes Source: https://citedby.eu/bots/yandexbot AI crawlers / YandexBot YandexBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer YandexBot is Yandex's main indexing robot, one of many Yandex robots with their own user agents. Yandex says some download pages for purposes other than indexing and may ignore restrictive rules written for all robots (User-agent: *). Its page gives no IP list for YandexBot and says to verify with a reverse DNS lookup. Operator Yandex Job Search Obeys robots.txt Yes Address list Reverse DNS On this page What YandexBot is What the vendor says What happens if you block YandexBot The robots.txt lines How to tell a real request from a fake one Compare with Googlebot Your own site Questions people ask Sources What YandexBot is YandexBot is listed by Yandex as the main indexing robot. Yandex runs many other robots with their own user agents (for images, video, advertising, mobile pages, favicons and more), listed on the same page. Yandex’s page gives no IP list for YandexBot; it tells you to verify requests with a reverse DNS lookup. Fact YandexBot Operator Yandex Job Search (main indexing robot) robots.txt token YandexBot Follows robots.txt Yes. Yandex’s page lists YandexBot as taking the general robots.txt rules (User-agent: *) into account. IP addresses Reverse DNS check Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “Some robots can disguise themselves as Yandex robots by indicating the relevant User Agent. You can check the authenticity of a robot using a reverse DNS lookup.” Source: Yandex Webmaster: How to check that a robot belongs to Yandex, read on 24 October 2026. What happens if you block YandexBot Yandex says a number of its robots download pages for purposes other than indexing and may ignore restrictive rules written for all robots (User-agent: *). To restrict those robots you need directives written specifically for them, for example User-agent: YandexCalendar. Yandex adds that a document the main Yandex robot cannot access will never be indexed or found in search results. The robots.txt lines To allow YandexBot: User-agent: YandexBot Allow: / To block YandexBot: User-agent: YandexBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. Yandex says User-agent: YandexBot applies only to the main indexing robot, while User-agent: Yandex applies to all Yandex robots. The user-agent string the vendor shows: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) How to tell a real request from a fake one Yandex says to run a reverse DNS lookup on the IP, check that the host name ends in yandex.ru, yandex.net or yandex.com, then confirm with a forward lookup; it also offers an IP address check tool. Compare with Googlebot Googlebot : Both are the main search indexing crawler of their operator; Yandex lists many separate robots, some of which ignore general robots.txt rules, and its page gives no IP list for YandexBot, only a reverse DNS check. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). YandexBot is not one of them: read your robots.txt for a YandexBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does YandexBot follow robots.txt? Yes. Yandex’s page lists YandexBot as taking the general robots.txt rules (User-agent: *) into account. Yandex says User-agent: YandexBot applies only to the main indexing robot, while User-agent: Yandex applies to all Yandex robots. How do I know YandexBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources Yandex Webmaster: How to check that a robot belongs to Yandex Yandex Webmaster: Using robots.txt Keep reading All AI crawlers compared Googlebot AI crawler settings by platform Can ChatGPT read my website? What decides it ## YouBot: what it does and what blocking it changes Source: https://citedby.eu/bots/youbot AI crawlers / YouBot YouBot: what it does and what blocking it changes By CitedBy · Page updated 25 October 2026. method Explanation written with AI from the official source cited on this page. Short answer YouBot is the web crawler that powers the You.com search engine, discovering and indexing pages for its results. You.com says it respects robots.txt, honors Crawl-delay and caches robots.txt for 30 minutes. User-agent: YouBot with Disallow: / blocks it from the whole site. Operator You.com Job Search Obeys robots.txt Yes Address list Listed on its page On this page What YouBot is What the vendor says What happens if you block YouBot The robots.txt lines How to tell a real request from a fake one Compare with MojeekBot Your own site Questions people ask Sources What YouBot is YouBot is the web crawler that powers the You.com search engine. It discovers and indexes web pages for You.com search results. You.com says X.X.X.X in the user agent stands for the Google Chrome version number. Fact YouBot Operator You.com Job Search robots.txt token YouBot Follows robots.txt Yes, robots.txt is cached for 30 minutes, so changes can take up to 30 minutes. IP addresses Listed on the vendor's page ( you.com/docs/youbot ) Vendor page read 24 October 2026 What the vendor says In the vendor's words (English original): “YouBot is the web crawler that powers the You.com search engine.” Source: You.com Docs: YouBot: You.com's Web Crawler, read on 24 October 2026. What happens if you block YouBot You.com says YouBot respects robots.txt and shows User-agent: YouBot with Disallow: / as the way to block it from the whole site. It also honors Crawl-delay and caches robots.txt for 30 minutes. The page does not describe any other You.com agents. The robots.txt lines To allow YouBot: User-agent: YouBot Allow: / To block YouBot: User-agent: YouBot Disallow: / Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. The user-agent string the vendor shows: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://you.com/docs/youbot; env:prod) Chrome/X.X.X.X Safari/537.36 How to tell a real request from a fake one You.com says requests originate from 68.67.112.0/24, hosts resolve as youbot-{octets-with-hyphens}.search.you.com, and requests carry Cloudflare Web Bot Auth signatures with public keys at you.com/.well-known/http-message-signatures-directory. Compare with MojeekBot MojeekBot : Both are search-engine crawlers that follow robots.txt; You.com supports Crawl-delay and signs requests with Web Bot Auth, while Mojeek does not support Crawl-delay. Your own site The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). YouBot is not one of them: read your robots.txt for a YouBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub. Questions people ask Does YouBot follow robots.txt? Yes, robots.txt is cached for 30 minutes, so changes can take up to 30 minutes. How do I know YouBot is really blocked? robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network. Sources You.com Docs: YouBot: You.com's Web Crawler Keep reading All AI crawlers compared MojeekBot AI crawler settings by platform Can ChatGPT read my website? What decides it ## Adobe Commerce: robots.txt is in Content, Design, Configuration Source: https://citedby.eu/platforms/adobe-commerce AI crawler settings by platform / Adobe Commerce Adobe Commerce: robots.txt is in Content, Design, Configuration By CitedBy · Page updated 27 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Adobe Commerce blocking ChatGPT? Adobe's pages name no AI crawler, so any block is a rule you add. Edit robots.txt in the Admin: Content, Design, Configuration, the Global row, then Search Engine Robots and the custom instruction box. Cloud projects have extra file-path steps. Where Content > Design > Configuration Section Search Engine Robots Cloud Extra steps AI names Not listed On this page Where the setting is on Adobe Commerce The one thing Adobe Commerce's pages say that the other platforms do not The steps The lines to use Gotchas Compare with Shopify Check your own site Questions people ask Sources Where the setting is on Adobe Commerce On the Admin sidebar go to Content, then Design, then Configuration. Adobe's SEO overview says to find the Global configuration in the first row of the grid, click Edit, and expand the Search Engine Robots section. The Edit Custom instruction of robots.txt file box is there, with a Reset to Default button. The one thing Adobe Commerce's pages say that the other platforms do not On Adobe Commerce on cloud infrastructure the robots.txt file has a location problem that the other platforms do not have. Adobe's implementation playbook says to add the content in the Admin, but the file system is read-only, so the file has to live under pub/media and a custom Fastly VCL snippet redirects /robots.txt there. For several sites in one project, the playbook names the files per site, for example domainone_robots.txt. For on-premises installs it says to write the files to /path/to/commerce/pub/media/ or /path/to/commerce/media, whichever is right for your installation. In the vendor's words: “Due to the read-only file system on Adobe Commerce on cloud infrastructure projects, you must specify the pub/media path before generating the file.” Source: Adobe Commerce Implementation Playbook: Best practices for configuring web crawlers, read on 24 October 2026. The steps On the Admin sidebar go to Content, then Design, then Configuration. Find the Global configuration in the first row of the grid and click Edit. Scroll down and expand Search Engine Robots. Set Default Robots to INDEX, FOLLOW, NOINDEX, FOLLOW, INDEX, NOFOLLOW or NOINDEX, NOFOLLOW. Adobe describes these as instructions to crawlers about indexing and following links. If needed, enter your own rules in the Edit Custom instruction of robots.txt file box. Reset to Default restores Adobe's instructions. Click Save Configuration, then open your store's /robots.txt to see the result. Adobe's playbook tips that you can view the auto-generated file there. The lines to use Adobe's own examples use User-agent: * for all crawlers. To allow or block one named crawler, add a group for its name: To allow: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Adobe's pages do not say whether text you type is added to or replaces its default instructions. Save, then read /robots.txt to confirm what is served. Gotchas Adobe's playbook says not to expose your Admin path in robots.txt, because that is a vulnerability for site hacking and potential loss of data, and to remove it. Adobe's default instructions start with User-agent: * and a list of Disallow lines such as /checkout/ and /catalog/, so check what you keep. On cloud, a knowledge base article dated 18 August 2026 says robots.txt content is generated from Admin configuration and stored in the database, with no physical file needed in that scenario, while the playbook describes the pub/media redirect. The two pages read differently, so confirm with Adobe Commerce Support which applies to your project. The playbook also says its advice covers the native Adobe Commerce storefront only, not headless or Adobe Experience Manager storefronts. Compare with Shopify Shopify : A theme template instead of an Admin box, and Support will not fix edits to it. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Why does my custom robots.txt not change on Adobe Commerce cloud? Adobe's knowledge base says the usual cause is that indexing by search engines is disabled, which stops custom robots.txt content being applied. Fix it in the Cloud Console or with the magento-cloud CLI restrict_robots setting. The article adds that Staging environments have indexing disabled by default and it cannot be enabled there. Does Adobe Commerce list AI crawlers for robots.txt? Not in the pages we read. They talk about web crawlers and bots, and their examples use User-agent: * only. Add each AI user agent you want to allow or block as its own group. Sources Adobe Commerce Implementation Playbook: Best practices for configuring web crawlers, read 24 October 2026 Adobe Commerce Admin guide: SEO overview, read 24 October 2026 Adobe Commerce knowledge base: Resolve robots.txt not updating or displaying default settings, read 24 October 2026 Keep reading All platforms compared Shopify AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Cloudflare: AI bot policies for Search, Agent and Training Source: https://citedby.eu/platforms/cloudflare AI crawler settings by platform / Cloudflare Cloudflare: AI bot policies for Search, Agent and Training By CitedBy · Page updated 27 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Is Cloudflare blocking ChatGPT? It can: Search, Agent and Training bots are set separately under Security Settings, then Configure AI bot policies, each as Block on all pages, Block on pages with ads, or Allow. Its page, last updated 1 July 2026, says updated defaults for new domains start on 15 September 2026. Where Security Settings Behaviors Search, Agent, Training Choices 3 per behavior Page updated 1 Jul 2026 On this page Where the setting is on Cloudflare Three behaviors, three choices and a date that has already passed The steps The lines to use Gotchas Compare with Shopify Check your own site Questions people ask Sources Where the setting is on Cloudflare In the Cloudflare dashboard go to Security Settings, then Configure AI bot policies. Each of the three behaviors gets its own setting. This works at the network level, so it applies whatever platform the site runs on. Three behaviors, three choices and a date that has already passed Cloudflare's presets are: Search, crawlers that collect or index your content to answer questions about it later; Agent, automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents; Training, crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers used for both Training and Search. Each preset blocks verified bots classified that way plus some unverified bots that behave similarly. The page, last updated 1 July 2026, announces new defaults for new domains from 15 September 2026: Training and Agent bots blocked on pages that display ads, Search remaining allowed, and mixed-purpose crawlers blocked by every configuration that blocks AI training, including the legacy Block AI bots option, which it marks as deprecating on that date. The page still used future tense when we read it on 21 October, five weeks after that date, so open your own dashboard and look at what is switched on rather than trusting either text. In the vendor's words: “Search: crawlers that collect or index your content to answer questions about it later.” Source: Cloudflare Docs: Block AI Bots, read on 21 October 2026. The steps Open your zone in the Cloudflare dashboard and go to Security Settings. Open Configure AI bot policies. For Search, Agent and Training pick Block on all pages, Block on pages with ads, or Allow. If you use the legacy Block AI bots option, check which mixed-purpose crawlers it now covers, because Cloudflare says it is deprecating. Test with the free AI-crawler access checker : a bot that robots.txt allows can still be refused here. The lines to use To allow: # Cloudflare settings are not robots.txt lines. # Choose Allow (do not block) for Search so answer engines can fetch pages. To block: # Cloudflare settings are not robots.txt lines. # Choose Block (on all pages) for Training; leave Search on Allow. Gotchas The risk for a shop that wants citations is the middle preset: Cloudflare defines Agent as chat fetch bots acting on a person's behalf, which is the job of fetchers like ChatGPT-User. Blocking Agent can stop an assistant opening your page when a person asks about it, even when your robots.txt allows it. Also note that Shopify says it does not recommend a proxy configuration in front of a Shopify store. Compare with Shopify Shopify : Says it handles bot management itself and advises against a proxy. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does Cloudflare's AI block override robots.txt? Cloudflare's page describes blocking by behavior at its network, so a bot can be refused there even where your robots.txt allows it. That is why a live request test matters. What is the difference between Search and Training on Cloudflare? Search crawlers index content to answer questions later; Training crawlers take content to train or fine-tune a model. Mixed-purpose crawlers are classed as Training. Is Block AI bots still available? Cloudflare marks it as deprecating on 15 September 2026. Its page was still written in future tense when we read it on 21 October 2026. Sources Cloudflare Docs: Block AI Bots, read 21 October 2026 Keep reading All platforms compared Shopify AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Framer: upload your own robots.txt with Static Files Source: https://citedby.eu/platforms/framer AI crawler settings by platform / Framer Framer: upload your own robots.txt with Static Files By CitedBy · Page updated 27 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Is Framer blocking ChatGPT? Framer generates a robots.txt for every published site; to allow or block AI crawlers you upload your own file through Static Files, with the path set to /. Framer says this is available on Pro and Enterprise plans, and your upload replaces the generated file. Where Static Files Plans Pro and Enterprise Default Auto-generated Custom file Replaces default On this page Where the setting is on Framer The one thing Framer's page says that the other platforms do not The steps The lines to use Gotchas Compare with Webflow Check your own site Questions people ask Sources Where the setting is on Framer In your project dashboard, select your domain, open the Files tab and upload your file with the path set to /. Framer's help page calls this Static Files. To see the live file first, add /robots.txt to your site's domain in a browser. The one thing Framer's page says that the other platforms do not Framer gives you no robots.txt editor. The file is generated for you, and the only way to change it is to upload a replacement, which is limited by plan. Framer's page says that if you need a custom robots.txt you can upload one using Static Files, and that the new file replaces the automatically generated one. The plan limit comes straight after. In the vendor's words: “This is available on Pro and Enterprise plans.” Source: Framer Help: Access your robots.txt file, read on 24 October 2026. The steps Write your robots.txt as a plain text file on your computer. Go to your project dashboard and select your domain. Open the Files tab. Upload your file with the path set to /. Open yourdomain.com/robots.txt in a browser to confirm the live file is yours. The lines to use To allow: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Gotchas Because your upload replaces the generated file, anything the generated file contained is gone unless you copy it first. Framer's page does not say what the generated file contains, so read yourdomain.com/robots.txt before you upload and keep any lines you want, such as a sitemap line. The page is about robots.txt in general and does not mention AI crawlers by name, so the user agents above are examples to check against each vendor's own page. It was last updated on 15 September 2026. Compare with Webflow Webflow : Rules are typed into a settings field rather than uploaded as a file, plus a Content-Signal header. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Can I edit the robots.txt file in Framer? Yes, by upload. Framer's page says to open the Files tab for your domain in the project dashboard and upload your file with the path set to /. It says this is available on Pro and Enterprise plans. Does Framer create a robots.txt automatically? Yes. Framer's page says it generates a robots.txt file for every published site, and that a custom file uploaded through Static Files replaces the default one. Sources Framer Help: Access your robots.txt file, read 24 October 2026 Keep reading All platforms compared Webflow AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Ghost: add robots.txt to your theme to allow or block AI crawlers Source: https://citedby.eu/platforms/ghost AI crawler settings by platform / Ghost Ghost: add robots.txt to your theme to allow or block AI crawlers By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Ghost blocking ChatGPT? Ghost serves a default robots.txt. To change it, save your own robots.txt at the root of your theme, then install and activate the new theme version. Ghost's help page says custom themes on Ghost(Pro) need the Creator plan or higher. Where Theme root File robots.txt Effect Replaces default Ghost(Pro) Creator or higher On this page Where the setting is on Ghost The one thing Ghost's page says that the other platforms do not The steps The lines to use Gotchas Compare with WordPress.com Check your own site Questions people ask Sources Where the setting is on Ghost There is no robots.txt field in the admin. Ghost's help page says the file lives inside your site's theme: write your robots.txt and save it to the root of the theme. Ghost's theme documentation says the same in one line: themes can include a robots.txt which overrides the default robots.txt provided by Ghost. The one thing Ghost's page says that the other platforms do not On Ghost the file travels with the theme, so the plan you are on decides whether you can change it. Ghost's help page says it requires you to be comfortable writing your own robots.txt and installing a new version of your theme. It then adds a plan limit for Ghost(Pro), Ghost's hosted service: installing a custom theme requires the Creator plan or higher. The page describes what happens next in these terms. In the vendor's words: “Once the new version of the theme is installed and activated, the theme's robots.txt will be used in place of Ghost's default robots.txt.” Source: Ghost Help: Modifying robots.txt, read on 24 October 2026. The steps Write your robots.txt file. Save it to the root of your theme. Install the new version of the theme, then activate it. Ghost says the theme's robots.txt is used in place of its default once the new version is installed and activated. On Ghost(Pro), check your plan first: Ghost says installing a custom theme requires the Creator plan or higher. The lines to use To allow: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Gotchas The theme's file is used in place of Ghost's default, not added to it. Ghost's pages do not say what the default contains, so open yoursite.com/robots.txt before you replace it and copy across any lines you want to keep, such as a sitemap line. The page also warns that this route needs you to be comfortable writing the file yourself, and it does not cover AI crawlers by name. Compare with WordPress.com WordPress.com : Its AI opt-out is a sharing setting, not a file you write. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Where do I put a custom robots.txt on Ghost? At the root of your theme. Ghost's help page says that once the new theme version is installed and activated, the theme's robots.txt is used in place of Ghost's default. Do I need a paid plan to edit robots.txt on Ghost(Pro)? Ghost's page says installing a custom theme on Ghost(Pro) requires a subscription to the Creator plan or higher. Because the file is part of the theme, that applies to a custom robots.txt too. Sources Ghost Help: Modifying robots.txt, read 24 October 2026 Ghost Docs: Theme structure, read 24 October 2026 Keep reading All platforms compared WordPress.com AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Netlify: block AI crawlers with robots.txt and User Agent Blocker Source: https://citedby.eu/platforms/netlify AI crawler settings by platform / Netlify Netlify: block AI crawlers with robots.txt and User Agent Blocker By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Netlify blocking ChatGPT? Not unless you add a rule. Netlify gives you two routes: a robots.txt file at the root of your site, or the User Agent Blocker extension, which blocks requests by user agent with an Edge Function. Netlify's guide says robots.txt alone is not enough, because some AI products ignore it. Where robots.txt in site root Edge block User Agent Blocker Activation One project at a time Fallback Edge Function On this page Where the setting is on Netlify The one thing Netlify's pages say that the other platforms do not The steps The lines to use Gotchas Compare with Cloudflare Check your own site Questions people ask Sources Where the setting is on Netlify There is no robots.txt editor in the dashboard pages we read. Netlify's developer guide says to state your rules in a robots.txt file "served from the root of your site". The separate docs page on blocking AI crawlers covers the User Agent Blocker extension, which you find under Extensions in your project's left navigation after installing it. The one thing Netlify's pages say that the other platforms do not Netlify's own guide treats robots.txt as the first of two layers, not the whole answer. It shows a robots.txt with one User-agent and Disallow block per bot, generated from a JSON list of names, and then adds an Edge Function that compares each request's User-Agent header with the same list and returns HTTP 401 to a match. It says some AI products do not respect robots.txt, and then adds that the user agent check has a gap of its own. In the vendor's words: “Sadly, some bots have been found to mis-report their names in their User Agent strings, so we can’t rely on that technique alone either.” Source: Netlify Developers: Blocking AI bots and controlling crawlers, read on 24 October 2026. The steps Write a robots.txt with a User-agent line and Disallow: / for each bot you want to block, and serve it from the root of your site. The guide generates the file from a list of names with a small build script, to avoid typos. Deploy the site. The guide notes that AI crawlers "should" honor the file, which is why it adds a second layer. For the extension: in your Netlify project dashboard select Extensions in the left sidebar, search for User Agent Blocker, select it, and select Install. This makes it available for all projects on your Netlify team. Go to the project where you want it active. You can only activate the extension for one project at a time. Under Extensions in the left navigation, select User Agent Blocker. Under Block User Agents, select the traffic to block. To target AI crawlers and bots, select all of the AI crawler options. Select Redeploy site and confirm with Save, then select Enable analytics and deploy site. Netlify says blocking starts working immediately once the deploy succeeds. The lines to use To allow: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Netlify's pages only show Disallow rules. The Allow lines above are standard robots.txt, not something Netlify documents. Gotchas Netlify says the extension's preset list "does not include all possible AI crawlers and AI bots", so a crawler missing from the list is not blocked. The same page lists SEO and search crawlers alongside the AI ones, so select only the AI options if you still want search traffic. The guide's example Edge Function answers with a 401 for every matching request, across the whole site. Compare with Cloudflare Cloudflare : A network-level switch that works whatever the site platform is, where Netlify's extension is per project. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Is robots.txt enough to block AI bots on Netlify? Netlify's guide says no. It says not all AI products respect robots.txt and recommends adding an Edge Function, or the User Agent Blocker extension, that blocks by user agent. It also says some bots mis-report their user agent, so the two layers together are its answer. Can I turn on the User Agent Blocker for every project at once? Not according to Netlify's docs. Installing it makes the extension available for all projects on your team, but you can only activate it for one project at a time. Sources Netlify Developers: Blocking AI bots and controlling crawlers, read 24 October 2026 Netlify Docs: Block AI crawlers and bots, read 24 October 2026 Keep reading All platforms compared Cloudflare AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## PrestaShop: robots.txt generator writes one rule set for all bots Source: https://citedby.eu/platforms/prestashop AI crawler settings by platform / PrestaShop PrestaShop: robots.txt generator writes one rule set for all bots By CitedBy · Page updated 27 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Is PrestaShop blocking ChatGPT? Its Robots file generation tool, under Shop Parameters, Traffic, SEO and URLs, writes directives for User-agent: * and names no AI crawler, so you add AI rules yourself. Clicking Generate robots.txt replaces any existing file, so add yours after generating. Where Traffic > SEO and URLs Button Generate robots.txt User-agent * only Regenerate Replaces the file On this page Where the setting is on PrestaShop The one thing PrestaShop's page says that the other platforms do not The steps The lines to use Gotchas Compare with Shopify Check your own site Questions people ask Sources Where the setting is on PrestaShop PrestaShop 8's user guide places it under Configuring your Shop Parameters, then Traffic, then SEO and URLs. The section is called Robots file generation. The page does not name a menu path inside the back office beyond that, and it shows "Last updated 4 years ago" at the foot. The one thing PrestaShop's page says that the other platforms do not PrestaShop does not give you a box to type rules into. It gives you a button that writes the whole file. The guide says the generator "simply creates a file with exclusion directives for files and directories that are not meant to be public", applied to all bots through the "User-agent: *" string. It also says the worst bots ignore the file, because it is "purely advisory". The consequence for AI crawlers is in the last two sentences of the section: regenerating wipes whatever you added. In the vendor's words: “Clicking on the "Generate robots.txt" button replaces any existing robots.txt file with a new one. Therefore, if you want to add your own rules, do it after PrestaShop has generated its version of the file.” Source: PrestaShop 8 User Guide: SEO and URLs, read on 24 October 2026. The steps Open SEO and URLs in the Traffic section of Shop Parameters. Go to Robots file generation. Click Generate robots.txt. This replaces any existing robots.txt file. Add your own rules after PrestaShop has generated its version. The page does not say how to edit the generated file, so use whatever file access your hosting gives you. Open /robots.txt on your shop to check that your rules are still there. Repeat step 4 every time you click the button again. The lines to use To allow: # Nothing to add: the generated file only lists files and directories that are not public. # To name one explicitly, add after the generated block: User-agent: OAI-SearchBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Gotchas The generated file speaks to every bot the same way, because it uses User-agent: * and nothing else, so PrestaShop's tool gives you no per-crawler choice. The guide says some bots you want to have full access, such as Google's, and others you do not, such as spam bots and content thieves, which is a decision you make in your own added groups. The guide does not mention AI crawlers by name at all, and it points to no setting or module for them. Compare with Shopify Shopify : A robots.txt.liquid template that keeps your additions, where PrestaShop's button overwrites them. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Will regenerating robots.txt in PrestaShop delete my AI crawler rules? Yes, if you added them to the generated file. PrestaShop's guide says the button replaces any existing robots.txt with a new one, and that you should add your own rules after it has generated its version. Does PrestaShop block AI crawlers by default? The guide does not say so. It describes a generated file with exclusion directives for non-public files and directories under User-agent: *, and it names no AI crawler. Sources PrestaShop 8 User Guide: SEO and URLs, read 24 October 2026 Keep reading All platforms compared Shopify AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Shopify: allow or block AI crawlers, and what robots.txt misses Source: https://citedby.eu/platforms/shopify AI crawler settings by platform / Shopify Shopify: allow or block AI crawlers, and what robots.txt misses By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Shopify blocking ChatGPT? Its default robots.txt allows crawlers that are not named, so a block is a rule you add in a robots.txt.liquid theme template. Shopify's help page says that affects only open-web discoverability: product data still reaches agentic storefronts such as ChatGPT and Microsoft Copilot through Shopify Catalog, independently of /robots.txt. Where robots.txt.liquid Support Unsupported by Shopify Catalog feed Not covered Effect Instant On this page Where the setting is on Shopify The one thing Shopify's page says that the other platforms do not The steps The lines to use Gotchas Compare with Squarespace Check your own site Questions people ask Sources Where the setting is on Shopify Online Store, then the theme's Edit code, then Add a new template and choose robots. Shopify says this is an unsupported customization: its Support cannot help with edits, and incorrect use "can result in loss of all traffic". The one thing Shopify's page says that the other platforms do not Shopify describes two layers of control. The network layer is handled by Shopify for stores on Shopify, and you do not need to act. The /robots.txt layer is yours through robots.txt.liquid. Its page then draws the line that matters for a shop: if you sell through agentic storefronts such as ChatGPT or Microsoft Copilot, your product data is syndicated through Shopify Catalog "independently of /robots.txt", and blocking AI crawlers at either layer "affects only open-web discoverability". To change what Catalog shares you use the agentic storefronts settings, not robots.txt. In the vendor's words: “Blocking AI crawlers at the /robots.txt or network layer affects only open-web discoverability. It doesn't stop your product data from being sent by Shopify Catalog to the agentic storefronts that you've activated.” Source: Shopify Help Center: Editing robots.txt.liquid, read on 21 October 2026. The steps From your Shopify admin go to Online Store. For the theme you publish, open the menu and choose Edit code. Click Add a new template, choose robots, then Create template. Keep the Liquid that outputs the default rule groups and add your own rules below it. Shopify recommends this over replacing the file with plain text, because plain text goes out of date. Save in the published theme. Shopify says changes are instant, but crawlers do not always react immediately. The lines to use To allow: # Nothing to add: Shopify's default robots.txt allows crawlers that are not named. # To name one explicitly, add under the default Liquid block: User-agent: OAI-SearchBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Gotchas Two details from the same page. A theme uploaded from the Themes section of the admin does not import robots.txt.liquid; ThemeKit or command-line changes keep it. And if your template outputs the default groups, delete it to pick up Shopify's current default, which can include rules your older copy lacks, such as Disallow: /search. Compare with Squarespace Squarespace : One checkbox instead of a template, and a fixed list of 26 names. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does blocking GPTBot on Shopify stop ChatGPT showing my products? Not by itself. Shopify says robots.txt and network blocks affect open-web discoverability only, and that Shopify Catalog sends product data to the agentic storefronts you have activated independently of /robots.txt. To control that feed, use Shopify's agentic storefronts settings. Can Shopify Support fix my robots.txt.liquid? No. Shopify's page says Support cannot help with edits to the file and suggests hiring a Shopify Partner. Do I need a Cloudflare proxy in front of Shopify? Shopify says bot management at the network layer is handled for stores on Shopify and that it does not recommend a proxy configuration in front of Shopify or support one. Sources Shopify Help Center: Editing robots.txt.liquid, read 21 October 2026 Shopify.dev: Customize robots.txt.liquid, read 21 October 2026 Keep reading All platforms compared Squarespace AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Squarespace: the AI crawler box blocks 26 bots but not ChatGPT Source: https://citedby.eu/platforms/squarespace AI crawler settings by platform / Squarespace Squarespace: the AI crawler box blocks 26 bots but not ChatGPT By CitedBy · Page updated 27 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Is Squarespace blocking ChatGPT? Only if you tick Settings, Crawlers, Block known artificial intelligence crawlers; unticked, all stay allowed. The box adds 26 named bots to robots.txt, including GPTBot, ClaudeBot and DuckAssistBot but not OAI-SearchBot, ChatGPT-User or PerplexityBot, and you cannot edit robots.txt yourself. Where Settings, Crawlers Names blocked 26 Edit robots.txt No Past training Not removed On this page Where the setting is on Squarespace The 26 names, and which ones are missing The steps The lines to use Gotchas Compare with Wix Check your own site Questions people ask Sources Where the setting is on Squarespace Open the Settings panel, click Crawlers, and tick Block known artificial intelligence crawlers. Squarespace's page says ticking the box updates your robots.txt file for the bots listed below, and that it may add more over time. The 26 names, and which ones are missing Squarespace's help page lists exactly 26 names: AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider and YouBot. Read against our own crawler pages, the box covers the training and dataset crawlers of OpenAI, Anthropic, Google, Apple, Meta and Common Crawl. It does not name OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot or Claude-User, so ticking it does not remove you from ChatGPT search. It does name DuckAssistBot and Amazonbot, which are answer-engine and product crawlers, so ticking it can also cost you DuckDuckGo's AI answers. In the vendor's words: “Checking the box updates your robots.txt file to tell the following bots not to crawl your site.” Source: Squarespace Help: Request that AI models exclude your site, read on 21 October 2026. The steps Open the Settings panel of your Squarespace site. Click Crawlers. Tick Block known artificial intelligence crawlers to block the 26 names, or leave it unticked to leave them all allowed. To ask for another bot to be added, contact Squarespace: your page's own text says you can request it. The lines to use To allow: # Squarespace sites are open to these crawlers while the box is unticked. # You cannot add lines yourself, so there is nothing to paste. To block: # Written by Squarespace when the box is ticked (excerpt): User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot Disallow: / Gotchas Squarespace states that ticking the box does not retroactively remove content already scraped from your site from AI model training data. It also says robots.txt is only a request that friendly crawlers have offered to obey. The excerpt above is our shortened illustration of the kind of lines the box produces; the exact file is whatever your site serves at /robots.txt, so open it and read it. Compare with Wix Wix : A real robots.txt editor, and an example list of four names. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Can I allow ChatGPT search but block training on Squarespace? Partly. Ticking the box writes a rule for GPTBot, and OAI-SearchBot is not among the 26 names. You cannot add lines of your own, so open your live /robots.txt to confirm what it says. Can I edit robots.txt on Squarespace? Squarespace's own page treats the Crawlers panel as the control and does not offer a robots.txt editor. Wix and Webflow do. Does the box stop Google from indexing my site? No. It is separate from the search engine crawlers option in the same panel. The AI box names Google-Extended and GoogleOther, not Googlebot. Sources Squarespace Help: Request that AI models exclude your site, read 21 October 2026 Keep reading All platforms compared Wix AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Vercel: the AI Bots ruleset in the Firewall (Log or Deny) Source: https://citedby.eu/platforms/vercel AI crawler settings by platform / Vercel Vercel: the AI Bots ruleset in the Firewall (Log or Deny) By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Vercel blocking ChatGPT? Not unless you change it. Vercel handles AI bots with a Firewall ruleset, not a robots.txt line: open Firewall, then Rules, and set the AI Bots Ruleset to Log or Deny. It is available on all plans and starts inactive, shown as Allow. Where Firewall, Rules Plans All plans Default Allow (inactive) Actions Log or Deny On this page Where the setting is on Vercel The one thing Vercel's page says that the other platforms do not The steps The lines to use Gotchas Compare with Cloudflare Check your own site Questions people ask Sources Where the setting is on Vercel From your project's dashboard, open Firewall in the sidebar, then Rules. In the Bot Management section you will find the AI Bots Ruleset rule. Vercel's docs say the ruleset is available on all plans. The one thing Vercel's page says that the other platforms do not Vercel's list of AI bots is not yours to maintain. The ruleset covers AI bots that crawl for training data, search purposes or user-generated fetches, and Vercel keeps the list current. The same page says the ruleset is inactive by default and labelled Allow in the dashboard, so matching traffic is not evaluated and reaches your application. The control is one action for the whole ruleset, Log or Deny, applied at the Firewall. In the vendor's words: “When new AI bots emerge, Vercel automatically adds them to its managed list and handles them according to your existing configured action without requiring any changes on your part.” Source: Vercel Docs: Bot Management, read on 24 October 2026. The steps From your project's dashboard, open Firewall in the sidebar. Open Rules in the sidebar. In the Bot Management section, select Log or Deny on the AI Bots Ruleset rule. Log records AI bot traffic without blocking it and is useful for monitoring. Deny blocks all traffic identified as coming from AI bots. When you make a change, a Review Changes button appears with the number of changes. Select Review Changes, check them, then select Publish to apply the changes to your production deployment. The lines to use Vercel's pages describe this as a Firewall setting, so there is no robots.txt line to add. The two states are: To allow: AI Bots Ruleset: Allow (the default, ruleset inactive) To block: AI Bots Ruleset: Deny Gotchas Deny blocks all traffic identified as coming from AI bots, and the ruleset's scope is training, search and user-generated fetches together, so it is not a way to block training bots while keeping search bots. Log is the safer first step. Vercel's page does not say how the ruleset relates to your robots.txt, so check what your own file says separately. For individual bots, the same page says you can block some or all of the bots in its verified bots directory with WAF custom rules that use the User Agent match condition or the Signature-Agent header. Compare with Cloudflare Cloudflare : Also a network-level switch, with three actions for each of Search, Agent and Training. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Does Vercel block AI bots by default? No. Vercel's docs say the AI Bots Ruleset is inactive by default, labelled Allow in the dashboard, and that matching traffic is not evaluated and reaches your application. Do I have to keep Vercel's list of AI bots up to date? No. Vercel says the list of known AI bots is automatically maintained and updated by Vercel, and that new bots are handled according to the action you have already set. Sources Vercel Docs: Bot Management, read 24 October 2026 Vercel Docs: WAF Managed Rulesets, read 24 October 2026 Keep reading All platforms compared Cloudflare AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Webflow: robots.txt rules and the Content-Signal header for AI Source: https://citedby.eu/platforms/webflow AI crawler settings by platform / Webflow Webflow: robots.txt rules and the Content-Signal header for AI By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Webflow blocking ChatGPT? That depends on the robots.txt rules you add in Site settings, SEO, Indexing; one help page says you need a Site plan or a paid Workspace plan. Webflow can also send a Content-Signal header that sets ai-train, search and ai-input separately, which it calls a proposed extension. Where Site settings, SEO Needs Site or paid Workspace Extra Content-Signal Status Proposed extension On this page Where the setting is on Webflow A header that splits training, search and AI answers The steps The lines to use Gotchas Compare with Wix Check your own site Questions people ask Sources Where the setting is on Webflow Go to Site settings, then SEO, then Indexing, add the robots.txt rule or rules you want, then click Save and publish your site. Webflow adds a sitemap line by default; a toggle removes it. A header that splits training, search and AI answers Besides robots.txt, Webflow's page says a site can serve a Content-Signal HTTP header, set in the same Indexing area, to give AI crawlers more granular instructions. Its example is Content-Signal: ai-train=no, search=yes, ai-input=no. Webflow explains the split: robots.txt controls whether bots can access your pages, while Content-Signal says whether content can be used for AI training, search indexing, or AI-generated responses, separately. Webflow adds the caveat itself: this is based on a proposed extension to RFC 9309 and is not yet an accepted standard. So it states a preference; it does not stop a crawler, and Webflow's page does not say which crawlers read it. In the vendor's words: “This is based on a proposed extension to RFC 9309 and is not yet an accepted standard.” Source: Webflow Help: Set robots.txt rules, read on 21 October 2026. The steps Open Site settings and go to SEO, then Indexing. Type the robots.txt rules you want in the rules field. Optionally set the Content-Signal header in the same area. Click Save and publish your site, then open yourdomain.com/robots.txt to check the live file. The lines to use To allow: User-agent: OAI-SearchBot Allow: / User-agent: * Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Content-Signal: ai-train=no, search=yes, ai-input=no Gotchas Webflow's page says your robots.txt is publicly visible, so do not rely on it to hide private pages, and that not all bots follow it. Compare with Wix Wix : A longer editing path and a four-name example. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Do I need a paid plan for robots.txt on Webflow? One Webflow help page says you need a Site plan or a paid Workspace plan to add a robots.txt file to your site. What is Content-Signal? A proposed extension to RFC 9309 that Webflow can send as a header: ai-train, search and ai-input, each set to yes or no. Does Webflow block bad bots for me? Its help page shows how to write a rule for a named bot. It says some bots, especially malicious ones, do not follow robots.txt. Sources Webflow Help: Set robots.txt rules, read 21 October 2026 Webflow Help: Request to block specific bots from visiting your site, read 21 October 2026 Keep reading All platforms compared Wix AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Wix: block AI crawlers in the Robots.txt Editor (four names) Source: https://citedby.eu/platforms/wix AI crawler settings by platform / Wix Wix: block AI crawlers in the Robots.txt Editor (four names) By CitedBy · Page updated 27 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer Is Wix blocking ChatGPT? You decide in the Robots.txt Editor. Wix lets you edit robots.txt: SEO & GEO in the dashboard, then Tools and settings, then Robots.txt Editor. Its help page pastes four names to block: CCBot, GPTBot, ChatGPT-User and BingAI. ClaudeBot, PerplexityBot and Google-Extended are not in that example, so add them yourself if you want them blocked. Where Robots.txt Editor Example names 4 Page-level nosnippet tag Support Not for edits On this page Where the setting is on Wix The example blocks four names, and one of them is not on Bing's list The steps The lines to use Gotchas Compare with Webflow Check your own site Questions people ask Sources Where the setting is on Wix In your site's dashboard go to SEO & GEO, scroll to Tools and settings, click Robots.txt Editor, then View File. Wix's separate editing guide calls this an advanced feature and says Wix Customer Care cannot help with changes you make to the file. The example blocks four names, and one of them is not on Bing's list Wix's help page says its list "includes some known AI crawlers, but you can always add more". The four it prints are CCBot, GPTBot, ChatGPT-User and BingAI. It leaves out the Anthropic, Perplexity, Google and Meta tokens that appear on our other pages. "BingAI" is the odd one: Microsoft's page on which crawlers Bing uses, read the same day, lists Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview and no BingAI. A rule for a name that no crawler uses does nothing, so treat Wix's list as a starting point and check it against each vendor's own page. Also note ChatGPT-User is the fetcher that opens a page for a person asking ChatGPT, so blocking it removes that route to your pages. In the vendor's words: “This list includes some known AI crawlers, but you can always add more as needed.” Source: Wix Help Center: Blocking AI Crawlers from Your Site, read on 21 October 2026. The steps Go to SEO & GEO in your site's dashboard. Scroll to Tools and settings and click Robots.txt Editor. Click View File, then click the field under Your robots.txt file. Add the directives under This is your current file, then Save Changes and Save. For one page only, open the page's SEO basics, Advanced SEO, Robots meta tag, and tick nosnippet. Wix says this prevents the content being used as direct input for AI overviews. The lines to use To allow: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / To block: User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / Gotchas Wix's FAQ says blocking AI crawlers in robots.txt may limit NLWeb functionality, because AI agents need access to your structured content. It also says some crawlers ignore robots.txt, so a block is not airtight. Compare with Webflow Webflow : Also a robots.txt editor, plus a Content-Signal header. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Will blocking AI crawlers hurt my Wix SEO? Wix says blocking specific AI crawlers should not significantly affect SEO, and recommends checking your analytics afterwards. Is BingAI a real crawler? It is not on Microsoft's current list of Bing crawlers, which names Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview. See the Bingbot page. Does nosnippet block AI crawlers? Wix describes it as preventing your content being used as a direct input for AI overviews. It is a page-level tag, not a crawler block. Sources Wix Help Center: Blocking AI Crawlers from Your Site, read 21 October 2026 Wix Help Center: Editing Your Site's Robots.txt File, read 21 October 2026 Microsoft Bing Webmaster help: Which Crawlers Does Bing Use?, read 21 October 2026 Keep reading All platforms compared Webflow AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## WordPress.com: AI opt-out is a privacy setting, not robots.txt Source: https://citedby.eu/platforms/wordpress-com AI crawler settings by platform / WordPress.com WordPress.com: AI opt-out is a privacy setting, not robots.txt By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is WordPress.com blocking ChatGPT? Not through robots.txt: its AI opt-out is a privacy setting. On WordPress.com, the AI opt-out is a privacy setting rather than a robots.txt edit. Its site visibility guide, last reviewed on 5 October 2026, says to opt out of sharing your public content with third parties, including AI models, you follow its Prevent third-party sharing guide. Where Privacy settings Not A robots.txt edit Guide reviewed 5 Oct 2026 Also MCP agent access On this page Where the setting is on WordPress.com Opting out of AI is a sharing setting here The steps The lines to use Gotchas Compare with Cloudflare Check your own site Questions people ask Sources Where the setting is on WordPress.com WordPress.com's visibility guide sends you to Settings, then Reading, for Site Visibility (Public, Coming Soon or Private), and to a separate Prevent third-party sharing guide for the AI opt-out. Opting out of AI is a sharing setting here WordPress.com's guide on controlling your site's visibility says: to opt out of sharing your site's public content with third parties (including AI models), visit the Prevent third-party sharing guide. That is a platform setting, separate from the robots.txt that search and AI crawlers read. The same page, last reviewed on 5 October 2026, also describes the opposite direction: you can connect external AI agents such as ChatGPT or Claude to read and edit your site's visibility settings by enabling MCP tool access. We could not read the body of the Prevent third-party sharing guide on 21 October: the page loaded with its title and a review date of 1 August 2024 but no steps, by a plain fetch and in a browser. So this page does not describe how that setting works; open the guide from your own dashboard. If you run self-hosted WordPress rather than WordPress.com, the file is yours to edit, as on any host. In the vendor's words: “To opt out of sharing your site's public content with third parties (including AI models), visit the Prevent third-party sharing guide.” Source: WordPress.com Support: Control your site's visibility, read on 21 October 2026. The steps Open Settings, then Reading, and check Site Visibility is Public if you want crawlers to see the site at all. Open WordPress.com's Prevent third-party sharing guide from the privacy settings help and follow it if you want to opt out of sharing content with third parties, including AI models. Open yourdomain.com/robots.txt and read what your site actually serves to crawlers. The lines to use To allow: # No edit to make on WordPress.com for the sharing opt-out. # Check what your site serves: # https://yourdomain.com/robots.txt To block: # On a self-hosted WordPress site, you add lines like these through your SEO plugin or the file: User-agent: GPTBot Disallow: / Gotchas A Coming Soon or Private site is not the same as an AI opt-out: it hides the site from everyone, search engines included, which is the opposite of what a shop that wants to be cited needs. Compare with Cloudflare Cloudflare : A network-level switch that works whatever the site platform is. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Is the WordPress.com AI opt-out the same as blocking GPTBot? WordPress.com describes it as opting out of sharing your public content with third parties, including AI models. It does not describe it as a robots.txt rule, so check your live robots.txt to see what crawlers are told. Can I edit robots.txt on WordPress.com? We did not find a robots.txt editor in the visibility guide we read, so we do not say either way. Look in your site's settings for your plan. What is MCP tool access? The guide says it lets external AI agents read and edit your site's visibility settings. It is about managing the site, not about crawlers. Sources WordPress.com Support: Control your site's visibility, read 21 October 2026 WordPress.com Support: Prevent third-party sharing, read 21 October 2026 Keep reading All platforms compared Cloudflare AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## WP Engine: robots.txt, 10-second default crawl-delay, Web Rules Source: https://citedby.eu/platforms/wp-engine AI crawler settings by platform / WP Engine WP Engine: robots.txt, 10-second default crawl-delay, Web Rules By CitedBy · Page updated 27 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Is WP Engine blocking ChatGPT? On a wpengine.com address, yes: it gets a disallow-all file. Otherwise a physical robots.txt in your site root overrides any file a plugin or theme generates, and the default file sets a crawl-delay of 10 seconds. To deny a bot by user agent, WP Engine points to its Web Rules Engine. Where SFTP, Yoast or Web Rules Crawl-delay 10 seconds default wpengine.com Disallow all Block by agent Web Rules Engine On this page Where the setting is on WP Engine The one thing WP Engine's page says that the other platforms do not The steps The lines to use Gotchas Compare with WordPress.com Check your own site Questions people ask Sources Where the setting is on WP Engine WP Engine's page gives two ways to make the file. With the Yoast SEO plugin installed, go to SEO, then Tools, choose the file editor and select Create robots.txt file. Or create a robots.txt file yourself and upload it with SFTP or the SSH Gateway to the root directory of your site. For blocking by user agent, the page sends you to access rules in the Web Rules Engine in the User Portal. The one thing WP Engine's page says that the other platforms do not WP Engine ships a default robots.txt, and the page says what is in it. Its note says the default crawl-delay for all sites is 10 seconds, and any environment on a wpengine.com address gets a file that disallows everything, to stop a site being indexed on the wrong domain before launch. Only environments using a custom domain will see changes made to robots.txt. A physical file you upload also takes priority over a file made by a plugin or theme. In the vendor's words: “This is what’s in the server’s default robots.txt file and can be overridden when you add a custom robots.txt file with one of the methods outlined in this article.” Source: WP Engine Support: Robots.txt and WordPress, read on 24 October 2026. The steps Create a file named robots.txt. The name must be lowercase and the extension .txt, not .html. Add your directives and save. For a plugin route instead, use Yoast SEO, Tools, then Create robots.txt file. Upload the file with SFTP or the SSH Gateway to the root directory of your site. Open /robots.txt on your live custom domain to check it. A copy on a wpengine.com address will not show your changes. To deny a bot at the server instead, open the Sites page in the User Portal, select the environment, click Web Rules in the secondary left menu (you may have to expand Advanced), choose Access rules and click Add Rule. Set Action to Deny, IP to All, and add a condition with Type Header, Name User-Agent, Operator Regex matches (~) and the user agent name as Value. Click Add Rule to save. The lines to use To allow: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / To block: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / WP Engine's page says directive values are case-sensitive and that globbing and regular expressions are not fully supported in robots.txt. Gotchas The page names no AI crawlers: its examples use BadBotName, Googlebot, bingbot and SEMrush. It also says robots do not have to follow robots.txt, and recommends a firewall such as Sucuri WAF or Cloudflare, or contacting Support, to stop bad actors before they reach the site. A request denied by a Web Rules access rule gets a 403 Forbidden response and is not counted towards billable visits, per the Web Rules Engine page. Your custom robots.txt overrides the default file, including its 10-second crawl-delay line, so add a Crawl-delay line yourself if you want to keep one. Compare with WordPress.com WordPress.com : An AI opt-out in privacy settings rather than a robots.txt file you write. Check your own site The free check in the box above reads robots.txt for the AI crawlers and sends one live request per crawler, so a rule that your platform's settings allow but a firewall refuses shows up. The full list of crawlers is on the AI crawler hub. Questions people ask Why does my staging site on WP Engine ignore my robots.txt? WP Engine says sites on an environment.wpengine.com address get a disallow-all robots.txt automatically, and only environments with a custom domain see changes to the file. Shared accounts can add a custom domain only to Production, so Staging and Development usually will not show your edits. Does a robots.txt plugin override a file I upload to WP Engine? No, the other way round. WP Engine says a physical robots.txt in the root of your site overwrites any dynamically generated file created by a plugin or theme. Sources WP Engine Support: Robots.txt and WordPress, read 24 October 2026 WP Engine Support: Web Rules Engine, read 24 October 2026 Keep reading All platforms compared WordPress.com AI crawlers compared AI visibility checklist for Shopify, WooCommerce and more ## Fail a deploy when AI crawlers can't read your site Source: https://citedby.eu/blog/ai-crawler-check-in-ci Guides / Crawlability How to fail a deploy when AI crawlers can't read your site By CitedBy · Page updated 25 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer Run the free citedby-check command or GitHub Action after each deploy. It loads your live site from outside, without JavaScript. It exits with 1 when robots.txt blocks one of five crawler names, the home page has very little text, or structured data is missing. It checks the home page and at most one product page. On this page What can shut AI crawlers out after a deploy? What does the check look at? Run it once from a terminal Add it to GitHub Actions What a pass does not tell you What is sent to citedby.eu? Prefer no dependency? Questions people ask Sources What can shut AI crawlers out after a deploy? Three changes can do it, and a look at the finished site shows none of them. A robots.txt written for staging ships to production with a blanket Disallow: /. Under User-agent: * it applies to every crawler that has no group of its own ( RFC 9309 ). A rewrite moves the page text into JavaScript. A visitor sees the full page, and a crawler that does not run JavaScript receives an almost empty shell. The check below reads the HTML as the server sends it, without running scripts, so it sees what such a crawler sees. A new CDN or firewall rule refuses the crawler. The check below cannot see this one, and the section on limits says how to test it. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update ( OpenAI ). Catching a bad file at deploy time shortens that exposure. How common is a robots.txt that shuts these crawlers out? In run 1 of our EU AI-Access Index, 15.3% of 10,426 popular EU websites blocked GPTBot. Of the same sites, 7.9% blocked at least one crawler that builds an AI answer index (OAI-SearchBot, Claude-SearchBot or PerplexityBot). The index records what each file asks AI crawlers to do. We treat blocking a training crawler such as GPTBot as a legitimate choice, and a later section shows how to keep one on purpose. Source: CitedBy AI-Access Index, run 1. robots.txt of 10,426 of 14,650 sites read on 23 Oct 2026. What does the check look at? citedby.eu loads robots.txt, your home page and llms.txt with a plain HTTP request, the way a crawler without a browser does. If the home page links to a product page, or the site has a Shopify product feed, it loads one product page too. The check runs from outside your pipeline. Source: README and code of citedby-check 0.1.0. Check What it looks at Result Crawlers in robots.txt Whether OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended or Bingbot is shut out of the home page. A server error (5xx) on robots.txt counts as shut out, as RFC 9309 says OK, Fix or Not checked Text in the HTML Words a crawler reads without running JavaScript: 200 or more is OK, 50 to 199 is Check, 1 to 49 is Fix. No readable text at all is Not checked OK, Check, Fix or Not checked Business data Organization, LocalBusiness or a similar business type as JSON-LD on the home page OK, Fix or Not checked Product data Product JSON-LD with price, availability and barcode on one product page, when the check finds one OK, Check, Fix or Not checked llms.txt Whether the site serves a real text file at /llms.txt: more than 50 characters, not an HTML page OK or Optional Google-Extended is a robots.txt switch only: no crawler fetches pages under that name, so the check reads whether robots.txt allows it. A missing llms.txt is reported as Optional and never fails a build. We do not expect the file to change what assistants answer ( Does llms.txt work? ). Run it once from a terminal You need Node 18 or newer and git. The package is not on the npm registry yet, so the command runs straight from GitHub: npx github:resaleiq/citedby-check yoursite.com To keep it installed, run npm install -g github:resaleiq/citedby-check, after which the command is citedby-check. The jq examples below assume it. Here is a real run on example.com from 23 October 2026. --yes skips the install prompt of npx, --source none keeps the run out of the count of CLI uses, and --no-color keeps the text plain. It exits with 1 because two checks need a fix. Run the same way on citedby.eu, it printed 4 of 4 checks passed. and exited with 0. $ npx --yes github:resaleiq/citedby-check example.com --source none --no-color AI-search readiness check for example.com citedby.eu loaded the site from outside: plain HTTP, no JavaScript. OK AI crawlers allowed in robots.txt No robots.txt found, so nothing is blocked. FIX Text in the page HTML Only 27 words are readable without JavaScript. Fix: render the homepage text on the server, so a crawler that does not run JavaScript still reads it. FIX Organization data (JSON-LD) No Organization or LocalBusiness data. Fix: add it as JSON-LD on the homepage with your name, logo, address or contact and social profiles, so assistants get your basic facts from one place. NOT CHECKED Product data (JSON-LD), Price in product data, Availability in product data, GTIN or barcode in product data No product page was found from the home page, so product data was not checked. OPTIONAL llms.txt No llms.txt. Optional hygiene: it is cheap to add, but we do not expect it to change what AI assistants answer. 1 of 3 checks passed. 2 to fix. This is a technical check. It does not tell you whether an AI assistant will name your site. The full audit crawls your site up to your plan's page limit and gives the exact fix for each problem: https://citedby.eu/?url=example.com $ echo $? 1 A line that starts with FIX fails the build: the exit code is 1. NOT CHECKED is never counted as a pass. No product page was found here, so nothing was scored. With no robots.txt file (HTTP 404) nothing is blocked, so the crawler check is OK. A robots.txt that answers a server error counts as blocked for all five names, as RFC 9309 says, and a robots.txt that cannot be loaded is NOT CHECKED. The exit codes are: 0: nothing at or above the level you set. 1: a check needs a fix. 2: wrong use, such as a missing address or an unknown option. 3: not measured: a bad address, the rate limit, a busy citedby.eu, a site that refused or did not answer, or no usable answer. A check that did not run is never reported as a pass. --fail-on warn also fails on Check, and --fail-on never exits 0 whenever the check ran, which gives you a report without a gate. --json prints the raw result, so you can list the failing checks in a script: citedby-check yoursite.com --json | jq -r '.checks[] | select(.status == "fail") | .id' Add it to GitHub Actions Create .github/workflows/ai-readiness.yml with the workflow below. Set url to the public address of the live site, and choose when it runs: weekly, after each deploy, or both. Open the run page after the first run and read the job summary and the annotations. name: AI-search readiness on: schedule: - cron: '0 6 * * 1' workflow_dispatch: jobs: check: runs-on: ubuntu-latest steps: - uses: resaleiq/citedby-check@v0 with: url: https://example.com The step fails when a check comes back as Fix. It writes the result to the job summary and marks each problem as an annotation. It also sets the outputs result ( pass, check, fix or not-measured ), passed, total, fix-count and check-count. The README suggests running it after a deploy or on a schedule. For a deploy trigger, see the workflow on our CI page : it uses deployment_status, which runs after Vercel, Netlify or Render reports a deploy. Annotations on the public run page of the Action's own smoke test against example.com. The errors come from a step with the default setting, the warnings from a step set to report only. A fifth item, a GitHub notice about runner images, is cropped. Run it against the live address after the deploy. citedby.eu checks from outside, so localhost, private network addresses and pages behind a login cannot be checked. The free check limits how often one site, and one connection, can be checked each hour. A run that hits the limit is reported as not measured with a warning, and it fails your build only when fail-on-error is true. What a pass does not tell you It covers the home page and at most one product page. A pass means a crawler can read the pages the check loads. It does not say that an AI assistant will mention or cite your site. It reads robots.txt rules and sends one kind of request. A CDN or firewall rule that refuses GPTBot by name is invisible to it. To test that, compare two requests as described in Is GPTBot blocked on my site?. The AI-crawler access checker sends a live request per crawler. A home page that sends no text at all is reported as Not checked. The check treats an empty answer as a fetch it cannot judge. A run with no failed check still exits with 0. To fail on that case too, test for the status unknown in the JSON: citedby-check yoursite.com --json --fail-on never | jq -e '[.checks[] | select(.status == "fail" or .status == "unknown")] | length == 0' If you block GPTBot on purpose, the crawler check reports Fix on every run. OpenAI treats its crawlers separately, so a site can allow OAI-SearchBot and disallow GPTBot ( OpenAI ). The command has no setting to ignore one crawler. With --fail-on never the command itself does not exit with 1, so jq decides. This filter fails only on checks other than a GPTBot-only block: citedby-check yoursite.com --json --fail-on never | jq -e '[.checks[] | select(.status == "fail") | select(.id != "crawlers" or ((.blocked - ["GPTBot"]) | length > 0))] | length == 0' What is sent to citedby.eu? The address, the language code and a tag ( cli, or action in the Action) go to citedby.eu. The request also names the tool and its version in the User-Agent header, and your IP address reaches citedby.eu as with any request. citedby.eu loads your site from its own servers. It adds one to a daily counter for the tag and one to its total count of checks. The command has no telemetry, no cookies and no account, and writes no files. A run tagged none is not counted under cli or action. The check still runs and still counts in the total. The README has the details. Prefer no dependency? The shell script on that CI page requests your site with each crawler's own user-agent string. It runs on your machine or in your CI, calls no AI model and sends nothing to CitedBy. It does not read JSON-LD or llms.txt. Questions people ask Does the check run my JavaScript? No. It loads robots.txt, the home page and llms.txt with a plain HTTP request and counts the words a crawler reads without running JavaScript: 200 or more is OK, 50 to 199 is Check, 1 to 49 is Fix. Does a pass mean AI assistants will cite my site? No. A pass means a crawler can read the pages the check loads, which are the home page and at most one product page. It does not say that an assistant will mention or cite your site. Can I block GPTBot and still pass the check? Not by default. The crawler check reports Fix on every run while GPTBot is blocked, because the command has no setting to ignore one crawler. The section on limits shows a jq filter that fails the build only on other checks. Does the check need an account? No. The command has no account, no cookies and no telemetry, and writes no files. The address, the language code and a tag go to citedby.eu, which loads your site from its own servers. Sources citedby-check on GitHub: README and source, version 0.1.0 CitedBy AI-Access Index, run 1 (robots.txt read on 23 October 2026) OpenAI: crawlers and user agents IETF RFC 9309: Robots Exclusion Protocol Keep reading Free AI crawler check for CI (shell script) Is GPTBot blocked on my site? AI-crawler access checker (free) EU AI-Access Index, run 1 Can ChatGPT read my website? ## Can ChatGPT read my website? What decides it Source: https://citedby.eu/blog/can-chatgpt-read-my-website Guides / Crawlability Can ChatGPT read my website? By CitedBy · Page updated 23 October 2026. method Explanation written with AI from the official source cited on this page. Short answer ChatGPT can read your website if three things hold: robots.txt does not block OAI-SearchBot, the page text is in the HTML your server sends rather than added by JavaScript, and your server or firewall does not refuse the crawler. The CitedBy audit tests the first two on every page. On this page The three gates How to check, step by step What a blocked page looks like to a crawler The gate the audit cannot see What to do with the result Questions people ask Sources The three gates ChatGPT search uses a crawler called OAI-SearchBot to find pages. OpenAI also runs GPTBot, which collects training data, and ChatGPT-User, which fetches a page when a person asks about it. OpenAI says each is controlled separately in robots.txt, so you can allow one and block another ( OpenAI crawler documentation ). The audit tests gates 1 and 3. Gate 2 needs your server logs. How to check, step by step Open yourshop.com/robots.txt in a browser. Look for User-agent: OAI-SearchBot, GPTBot or * followed by Disallow: /. A blanket Disallow: / under User-agent: * blocks every crawler that has no rule of its own. Run the free check in the box above. It reads robots.txt for OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended and Bingbot and tells you which are blocked. Look at the word count it reports for the page HTML. Your homepage should have hundreds of words there, including your brand name and what you sell. Open the page, choose "View page source" and search for your product name. If it is missing, the page builds its text with JavaScript and a crawler that does not run scripts sees nothing. Check that Organization data (JSON-LD) is present, so assistants get your name, logo and contact details from one place. A real run on bbc.com, 18 Oct 2026. The BBC blocks these crawlers on purpose; for a shop that wants to be named, a result like this is the problem to fix. What a blocked page looks like to a crawler Some shop themes and website builders send an almost empty page and fill it in with JavaScript. A visitor sees a full product page. A crawler that does not run JavaScript receives a shell. Illustration with invented example values. Some assistants do run JavaScript and some do not, so the safe position is to have your brand and product text in the HTML itself. The gate the audit cannot see robots.txt is a request, and your hosting is the enforcement. A firewall or CDN setting can answer a crawler with an error even when robots.txt allows it. The audit does not test this. Look in your server or CDN logs for requests from OAI-SearchBot and see which status codes came back; a run of 403 responses is the sign. What to do with the result robots.txt blocks a crawler you want: remove the matching Disallow rule. OpenAI says changes can take about a day to take effect. Few words in the HTML: render the homepage and product text on the server. No Organization data: add it as JSON-LD on the homepage. All clear: being readable is the entry ticket, not the result. Whether an assistant names your shop is a separate question, which the audit measures by asking real buyer questions. Questions people ask Is GPTBot the same as ChatGPT search? No. GPTBot collects data that may be used to train models. OAI-SearchBot is the crawler for ChatGPT search results. You can block GPTBot and still allow OAI-SearchBot. If I block GPTBot, will ChatGPT stop mentioning my shop? Not by itself. Search results depend on OAI-SearchBot. Blocking only GPTBot is a common choice for shops that do not want their content used for training. Does adding llms.txt fix this? It is optional. We do not expect it to change what assistants answer, and the audit marks it as optional. How long until a change shows up? OpenAI says it can take about 24 hours for its systems to adjust after you change robots.txt. Sources OpenAI: crawlers and user agents CitedBy site audit Keep reading AI crawlers: what each one does and how to allow or block it AI visibility audit: what it is and what to check Who ChatGPT linked, by niche A sample report Findable check: can search engines find your pages? (free) How to get cited by ChatGPT, Perplexity and Claude ## ChatGPT shows my old price: find the source and fix it Source: https://citedby.eu/blog/chatgpt-wrong-price Guides / Wrong facts ChatGPT shows my old price. How to find where it read it and fix it By CitedBy · Page updated 4 November 2026. 4 sources · method Explanation written with AI from the official source cited on this page. Short answer ChatGPT shows an old price when it read that price from a stale source: your own page, a product feed, or another website. Search for the old price in each place, fix the source, then ask again twice in a new chat. Update your feed whenever a sale starts or ends. On this page Why an assistant quotes a price you changed Step 1: write down the wrong answer Step 2: look for the old price on your own page Step 3: check your product feed Step 4: search for the old price on other websites Step 5: wait, then re-ask Can the audit find this? Questions people ask Sources Why an assistant quotes a price you changed An assistant answers from what it can find: pages it fetched, feeds a merchant submitted, and third-party pages that mention you. If one of those still carries the old figure, the answer can carry it too. That makes a wrong price a data problem on a page you control or can contact, which is why it can be fixed. The fix happens at the source, not in the chat. A reader on r/ecommerce described exactly this on 22 September: ChatGPT recommended a bundle that had been discontinued a year earlier, at the old $149 price ( thread ). It is one report, not a measured rate, so treat it as an illustration of the pattern. Check the three sources in this order: your page, your feed, then other sites. Step 1: write down the wrong answer Open a new chat, with web search on if the assistant offers it, and ask: "What does [product] from [shop] cost?" Copy the answer, the date, the assistant and any link it cites. A cited link is the best clue, because it names the page that fed the answer. Ask a second time in another new chat. If the two answers differ, log both, since answers vary between runs. Step 2: look for the old price on your own page Open the product page and view the page source (right click, View page source). Search for the old price, for example 49.00. Try with and without the currency sign. Find the application/ld+json block and read the offers price. Product markup that was never updated is a common reason for a stale figure, and Google says the price in markup and feed must match the landing page ( Google Merchant Center help ). Look for a second price on the page: a "was" price, a bundle price, a variant that loads a different amount. Fetch the page the way a crawler does, with no JavaScript. If the price only appears after scripts run, a crawler may not see the final figure. Google's help page also warns that slow pages can make crawlers miss the final price. Step 3: check your product feed If you sell through Google Shopping, a marketplace or ChatGPT's shopping feed, the feed is a second copy of your price. OpenAI's product feed specification asks for a price in the form 79.99 USD, one row per purchasable item or variant, and says to update the feed when a sale starts or ends ( OpenAI product feed spec ). A sale that ended on your site but stayed in the feed is the classic mismatch. Google describes the same trap for its own feed: when the feed and the landing page disagree, the fix is to match them, resubmit through feed upload, the Content API or your shop platform, and then request a review, which Google says takes 7 business days. If Google has already disapproved your products as a precaution and the review fails, you must wait 7 more business days before you can request another. If your account only had a warning, Google reviews it automatically once the warning deadline passes ( Google Merchant Center help ). OpenAI's page does not describe a way to dispute data already ingested, so the steps here cover what you control: your feed, your pages and your robots.txt. Step 4: search for the old price on other websites Search the old price together with your brand name and product name. Look at review sites, price lists, comparison pages, old blog posts and your own press releases. Ask the owner of each page to update it. Where you cannot, publish a current page on your own site that states the price with the date, so the newest source is also the clearest one. Step 5: wait, then re-ask OpenAI says a robots.txt change can take about 24 hours to reach its search crawler ( OpenAI crawler documentation ). It publishes no schedule for refreshing a page it has already read, so track it instead of waiting for a date. Re-ask the same prompt twice, in new chats, once a week, and log each answer in a table. Fix the source, not the chat. Then log every re-test. Date Assistant Prompt Price shown Link cited Matches your page? 2026-10-18 ChatGPT, search on What does [product] from [shop] cost? Can the audit find this? The CitedBy audit crawls your pages and tests whether AI crawlers can reach them and read your product data. If your price page is blocked or empty to a crawler, no fix to the price can help, so test that first. Questions people ask Why does ChatGPT show an old price for my product? It probably read the old figure from your page, your product feed or another website that still carries it. Search the old price in each place and look at the link it cites, if there is one. How long until ChatGPT shows the new price? OpenAI publishes no refresh date. It says robots.txt changes take about 24 hours to reach its search crawler; for pages already read it gives no schedule. Re-ask weekly and log the answers. Can I ask OpenAI to change a wrong price? OpenAI's feed specification does not describe a way to correct data already ingested, so fix the sources instead and re-test. Check OpenAI's help centre for any report option that exists when you read this. Does a price in schema markup matter? Yes, when it is stale. Google says markup and feed price should match the landing page, and a markup block that was never updated is a common mismatch to look for. Sources OpenAI: product feed specification OpenAI: crawlers and user agents Google Merchant Center: inaccurate price due to feed and landing page mismatch r/ecommerce, 22 September 2026: ChatGPT recommends a discontinued bundle at the old price CitedBy site audit Keep reading Check what AI says about your brand, 20 prompts Can ChatGPT read my website? A sample report Profound vs Peec vs Otterly compared ## What does ChatGPT say about your brand? 20 prompts Source: https://citedby.eu/blog/check-what-ai-says-about-your-brand Guides / Brand checks What does ChatGPT say about your brand? 20 prompts to check it by hand By CitedBy · Page updated 11 November 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer To check what ChatGPT says about your brand, ask it 20 prompts in four groups: your brand, your category, comparisons with rivals, and facts such as price and shipping. Ask each twice in a new chat, log whether it names you, links you and gets the facts right, and count the results. On this page What to prepare The 20 prompts How to score each answer What to do with your log The faster way Questions people ask Sources What to prepare You need a free account on each assistant you care about, a spreadsheet, and about an hour. Open a new chat for every prompt so earlier answers do not colour the next one, and turn web search on if the assistant offers it, because an answer built from live pages can differ from one without. Write down which assistant, which setting and the date. Answers change between runs, which is why every prompt gets asked twice. Fill the brackets with your own words: [category] is what you sell (for example "natural latex mattresses"), [country] is where you ship, [brand] is your shop name and [rival] is a shop you compete with. The 20 prompts Group A: your brand (does it know you?) Prompt Ask this P1 What is [brand]? P2 Is [brand] a good place to buy [category]? P3 What do customers say about [brand]? P4 Where is [brand] based and where does it ship? P5 What does [brand] sell and what is it best known for? Group B: your category (are you named?) Prompt Ask this P6 Where can I buy [category] online in [country]? P7 Which online shop is best for [category] in [country]? P8 Which shop delivers [category] fast in [country]? P9 What should I buy as a gift: [category], and from where in [country]? P10 Which small or independent shops sell good [category] in [country]? Group C: comparisons (who beats you?) Prompt Ask this P11 [brand] or [rival] for [category]? P12 What are alternatives to [rival] for [category] in [country]? P13 Is [brand] cheaper than [rival]? P14 Which shops are similar to [brand]? P15 Why would I choose [brand] over a big marketplace for [category]? Group D: facts (is it right?) Prompt Ask this P16 How much does [brand] charge for [your best-selling product]? P17 Does [brand] still sell [a product you discontinued]? P18 What is [brand]'s return policy? P19 How much is shipping from [brand] to [country]? P20 Does [brand] have a physical shop? Copy this layout into a spreadsheet. The rows shown are invented examples. How to score each answer Run the prompt and save the full answer text and every link. Mark named when your brand name or your domain appears in the answer text or in its links. If you appear as a link only, still mark it, and note it. Mark linked when a link goes to your own domain. Copy every other shop that is named. Rivals that appear again and again are your real competition in that assistant. For groups A and D, check every fact against your own site and mark the ones that are wrong or old. Run the same prompt once more in a new chat and log it as run 2. Do not round up. A shop named in 1 of 2 runs is named in 1 of 2 runs, not "recommended". What to do with your log Named in few group B answers: the assistant does not connect your shop to your category. Look at which pages it links for those answers; those are the pages it uses for your category. Wrong facts in group D: fix the source it read. Prices and policies on your own pages come first, then third-party listings. Rivals named again and again: open the pages the assistant linked for them and note what they contain that yours lacks. Nothing named anywhere, including your brand: check first that assistants can read your site at all ( Can ChatGPT read my website? ). OpenAI says sites that are opted out of OAI-SearchBot, its search crawler, "will not be shown in ChatGPT search answers, though can still appear as navigational links" ( OpenAI: Overview of OpenAI Crawlers ). Google says that to be a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet ( Google Search Central: AI features and your website ). The faster way Twenty prompts across five assistants is 200 manual runs. A CitedBy audit works on your site instead: it crawls up to your plan's page limit, shows which pages assistants cannot reach or read, and gives the exact fix for each. The audit starts at €9. Questions people ask Do I need ChatGPT Plus to run these prompts? No. A free account is enough to see what a free user sees. Paid plans can answer differently, so note which plan you used. Why ask every prompt twice? Answers vary between runs. Two runs show whether a result is stable; one run can be luck. Should I ask for my brand by name in every prompt? No. Groups B and C test whether you are named when the buyer does not know you yet, which is the case that brings new customers. Groups A and D test what is said once they do. How often should I repeat this? After each change to your site and about once a month. Log the date every time so you can compare. Sources CitedBy sample audit: method, questions and CSV About CitedBy: how we count OpenAI: Overview of OpenAI Crawlers Google Search Central: AI features and your website Keep reading A sample report Can ChatGPT read my website? Who ChatGPT linked, by niche Profound vs Peec vs Otterly compared ## Cloudflare blocking AI crawlers? How to allow them (2026) Source: https://citedby.eu/blog/cloudflare-ai-crawlers Guides / Crawlability Cloudflare is blocking AI crawlers on my site. How to allow GPTBot and ClaudeBot By CitedBy · Page updated 21 October 2026. 4 sources · method Explanation written with AI from the official source cited on this page. Short answer In the Cloudflare dashboard, open your domain, then AI Crawl Control, then the Security tab. Find each crawler, such as GPTBot or ClaudeBot, and choose Allow in the Actions column. Blocking creates a WAF rule, so removing it means switching the crawler back to Allow. Then run a check from outside. On this page Why Cloudflare can block AI crawlers without you touching robots.txt Allow a crawler in Cloudflare, step by step Which crawler to allow for which goal How to confirm it worked Should you allow everything? Questions people ask Sources Why Cloudflare can block AI crawlers without you touching robots.txt Cloudflare sits in front of your shop and can refuse a crawler before your server sees the request. That is why a robots.txt that welcomes every bot does not settle the question. When you choose Block for a crawler in AI Crawl Control, Cloudflare creates or updates a WAF custom rule on your zone to enforce it ( Cloudflare docs ). Defaults are moving too. Help Net Security reports that from 15 September 2026 the default settings for new domains block Training and Agent crawlers on pages that display ads, while Search crawlers stay allowed. Cloudflare said it would notify customers before the change. That date has passed, so read your own dashboard instead of guessing which default your domain has ( Help Net Security ). Allow a crawler in Cloudflare, step by step Log in to the Cloudflare dashboard and select your account and domain. Open AI Crawl Control, then the Security tab. Find the crawler in the table, for example GPTBot or ClaudeBot. In the Actions column choose Allow. The same menu offers Block. If you also want robots.txt rules honoured, enable Enforce robots.txt. Cloudflare says this stays available when you allow a crawler. Run the free check below on your homepage, then repeat the two-request test from our GPTBot guide. The setting sits under AI Crawl Control, not under the general firewall rules. Two details from Cloudflare's documentation. A blocked crawler can be answered with a 403 or a 402 code and a custom message, and changing that response is available on paid plans only. Free plans can still allow and block. Which crawler to allow for which goal OpenAI and Anthropic each publish three crawlers with different jobs. Allowing the search crawler is the usual way to be findable, and you can treat the training crawler as a separate decision. Job OpenAI Anthropic Collect content that may train models GPTBot ClaudeBot Improve search results OAI-SearchBot Claude-SearchBot Fetch a page because a person asked ChatGPT-User Claude-User Same three jobs at both companies. Allow or block them one by one. Anthropic says its bots follow robots.txt and support Crawl-delay. It publishes the IP ranges its crawlers use at claude.com/crawling/bots.json, and warns that blocking by IP address alone may stop the bot from reading your robots.txt ( Anthropic help centre ). OpenAI publishes its crawler list and IP ranges in its documentation ( OpenAI ). How to confirm it worked Cloudflare changes are not proof until a request gets through. Three checks, in order: Run the free check in the box above. It reads robots.txt for the main AI crawlers and tells you if your page answers a plain request. Send two requests with curl, one normal and one with a crawler user-agent, and compare the status codes. The commands are in our GPTBot guide. Look at your server or Cloudflare logs a day later for the crawler's name and the status codes it received. If the dashboard says Allow but the second request still returns 403, look for a second rule: a security plugin, a hosting firewall or an older WAF rule can refuse the same traffic. Should you allow everything? Not necessarily. Blocking training crawlers while allowing search crawlers is a legitimate setup, and Cloudflare's own defaults lean that way for pages with ads. What costs shops is blocking by accident. A site that blocks the search crawlers cannot be fetched for those search features, so decide per crawler and write down why. Questions people ask Where is the AI crawler setting in Cloudflare? Open your domain, then AI Crawl Control, then the Security tab. Each crawler has an Actions column with Allow and Block. How do I unblock ClaudeBot in Cloudflare? Find ClaudeBot in the crawler table on the Security tab and choose Allow. Anthropic also lists Claude-SearchBot and Claude-User as separate crawlers, so check those rows too. Does allowing a crawler in Cloudflare override my robots.txt? No. Cloudflare says you can still enforce robots.txt after allowing a crawler. A Disallow rule in your robots.txt still tells that crawler to stay away. Is the 15 September 2026 default change affecting my shop? Help Net Security describes it as a change to the defaults for new domains. If your domain is older, your existing settings stand until you change them. Check the Security tab to see what applies. Sources Cloudflare: manage AI crawlers Help Net Security: Cloudflare changes AI crawler access rules, 2 July 2026 Anthropic: does Anthropic crawl data from the web, and how can site owners block the crawler OpenAI: crawlers and user agents, IP ranges CitedBy free check, method note Keep reading Is GPTBot blocked on my site? Can ChatGPT read my website? The free crawler check Findable check: can search engines find your pages? (free) What changed: log of AI-crawler and search changes ## Does llms.txt work? What the data says (2026) Source: https://citedby.eu/blog/does-llms-txt-work Guides / AI visibility basics Does llms.txt work? What two studies and Google's own documentation say By CitedBy · Page updated 4 November 2026. 5 sources · method Explanation written with AI from the official source cited on this page. Short answer So far the evidence says llms.txt does not move AI citations. Ahrefs found 97% of valid llms.txt files got zero requests in May 2026, SE Ranking found no clear link to citations across about 300,000 domains, and Google says you do not need AI text files. Adding one is cheap, but do not expect traffic from it. On this page What llms.txt is What the data says What Google says So should you add one? What to do instead first Questions people ask Sources What llms.txt is llms.txt is a proposal by Jeremy Howard, published in September 2024: a Markdown file at /llms.txt with a title, a short summary and lists of links, meant to give AI agents a concise map of a site ( llmstxt.org ). It is a community proposal: no search engine or AI lab has adopted it as a standard. The proposal's own page lists adoption by documentation platforms and says OpenAI, Anthropic and Gemini publish the file for their own developer docs. That shows people write it. It does not show that assistants read yours when they answer a shopper. What the data says Figures as reported by Ahrefs. Most requests came from audit tools and unknown bots, not assistants. Does anything fetch it? Ahrefs checked 137,210 domains with its web analytics in May 2026. It found that 97% of valid llms.txt files received zero requests that month. Of the requests that did arrive, 19.5% came from named AI tools. SEO audit tools, unknown bots and general crawlers made up much of the rest. Ahrefs also reports that no AI bot went looking for the file on sites that did not have one ( Ahrefs study ). Does it change citations? SE Ranking analysed about 300,000 domains and published in November 2025 that having llms.txt showed no clear connection to how often a domain is cited by AI, with 10.13% of domains having the file ( Search Engine Journal on the SE Ranking study ). Their own wording is that it "doesn't seem to directly impact AI citation frequency. At least not yet." How common is it on launched sites? In our launch crawl of 300 Show HN sites and 70 Lovable builds (20 and 21 October 2026, CitedBot, no JavaScript), 34% of Show HN sites (100/293) and 18% of Lovable sites (12/66) served an llms.txt file. In the same crawl, 16% of Show HN homepages and 74% of Lovable ones sent CitedBot under 100 words of text in their HTML. We did not measure citations, so the crawl shows how often the file exists, not what it does. Both studies are correlation or log data from outside parties, run on different samples with different methods. They support "no measurable benefit so far", not "it can never matter". What Google says Google's documentation on AI features states that there are no additional requirements to appear in AI Overviews or AI Mode, and that you do not need to create new machine-readable files, AI text files or markup to appear in them ( Google Search Central: AI features ). For ChatGPT, OpenAI documents the crawlers it uses (OAI-SearchBot, GPTBot, ChatGPT-User) and says robots.txt controls them; it does not describe llms.txt there ( OpenAI crawler documentation ). So should you add one? Your site Add llms.txt? Why Online shop, no developer docs Optional, after the basics The data shows no citation benefit, so fix crawler access and on-page facts first Software with public docs or an API Reasonable Coding agents and audit tools do fetch it, and Ahrefs saw agents among the requesters You already have one Leave it, keep it accurate A stale file that lists dead pages helps nobody If you add one, keep it short: your name, one sentence on what you sell and where you ship, and links to your main category, shipping and returns pages. It takes minutes. What to do instead first Run the free check in the box above. It tests whether AI crawlers can fetch your page and read it, and it reports whether an llms.txt exists as a hygiene item, without treating it as a ranking factor. Put price, shipping and returns in the page text, not only in images or scripts. Ask an assistant 20 buyer questions twice each and log who gets named ( the method ). Questions people ask Does ChatGPT read llms.txt? There is no published evidence that it does. OpenAI documents its crawlers and robots.txt but does not describe llms.txt, and Ahrefs found most requests to llms.txt files came from other kinds of bots. Does llms.txt help me get cited by AI? Not on current evidence. SE Ranking found no clear link across about 300,000 domains, and Google says no AI text file is needed for its AI features. Can llms.txt hurt? It can mislead if it is stale or lists pages you removed. Keep it short and accurate, or do not publish one. Is llms.txt the same as robots.txt? No. robots.txt tells crawlers what they may fetch. llms.txt is a proposed summary file for AI agents that no major search engine requires. Sources llmstxt.org: the llms.txt proposal Ahrefs: llms.txt study, 137,210 domains, May 2026 Search Engine Journal: SE Ranking study of about 300,000 domains, 20 November 2025 Google Search Central: AI features and your website OpenAI: crawlers and user agents CitedBy site audit Keep reading llms.txt checker (free) Best GEO and AI visibility tools compared Can ChatGPT read my website? GEO vs AEO vs SEO audit A sample report Findable check: can search engines find your pages? (free) What changed: log of AI-crawler and search changes ## GA4: track traffic from ChatGPT and AI assistants Source: https://citedby.eu/blog/ga4-ai-referral-traffic Guides / Measurement Is AI traffic converting? How to set up GA4 to see visits from ChatGPT and other assistants By CitedBy · Page updated 21 October 2026. method Explanation written with AI from the official source cited on this page. Short answer Create a custom channel group in GA4 with an AI assistants channel whose Source matches a regex of the assistants you see in your Traffic acquisition report. Move it above Referral, because GA4 uses the first matching channel. Then compare conversions by channel. Visits without a referrer stay under Direct. On this page What GA4 shows by default, and why AI visits hide Build an AI assistants channel in GA4 A starter regex, and how to check it What GA4 cannot tell you Make the next step visible Questions people ask Sources What GA4 shows by default, and why AI visits hide GA4 sorts every session into a channel. Visits from an assistant can end up in the Referral channel, mixed with links from blogs and directories, or under another channel, so you cannot see them as a group. A custom channel gives them their own row. Google's documentation describes how: a custom channel group can be applied to your reports retroactively, so history is relabelled as soon as you save it ( Google Analytics Help ). You need the Editor role or above on the property to create one. Standard properties can hold 2 custom channel groups, with up to 50 channels each, according to the same page. Build an AI assistants channel in GA4 In GA4, open Reports, then Acquisition, then Traffic acquisition. Change the dimension to Session source / medium. Scan the list for assistant names such as chatgpt, perplexity, claude, gemini or copilot. Write down the exact sources you see. Open Admin, then Data display, then Channel groups, and choose Create new channel group. Choose Add new channel, name it AI assistants, and add a condition group: Source, matches regex, then paste your list of sources. Save the channel, choose Reorder, and drag AI assistants above Referral. Click Apply. Reopen Traffic acquisition with your new channel group as the dimension and compare sessions, engaged sessions and conversions. Order matters: GA4 gives each session to the first channel it matches. The order step is the one people miss. Google says traffic is included in the first channel whose definition it matches, given the current order. If Referral sits above your new channel, assistants never reach it. A starter regex, and how to check it Download the file with the steps and the regex. It starts from the names of the well-known assistants: (chatgpt|openai|perplexity|claude|gemini|copilot) Treat it as a starting guess, not a verified list. Hostnames change and assistants launch, so what matters is what appears in your own report. Add the sources you really see, and remove any that match something that is not an assistant. Run the check at least once a quarter. Match what your own report shows, not a copied list. What GA4 cannot tell you Only visits that carry a referrer can be labelled. A person who copies your address from an answer and pastes it into a browser arrives as Direct. A channel in GA4 therefore shows a lower bound of AI-driven visits, not the total. Say so when you report the number to anyone who will act on it. Small numbers can be noise. Compare conversion rate and revenue per session against Organic Search before drawing any conclusion, and do not extrapolate from a week of data. Make the next step visible The channel tells you whether assistants send visitors. It does not tell you whether they name your shop. The free check shows whether AI crawlers can read your pages, and the audit asks real buyer questions to see who gets named. Questions people ask How do I see ChatGPT traffic in GA4? Open Traffic acquisition with Session source / medium and look for chatgpt in the source column. For a standing view, create an AI assistants channel as above. Why does my AI traffic show as Direct? When the visitor's browser sends no referrer, GA4 has nothing to label. Pasting a link or opening it from an app without referrer information gives that result. Does a custom channel group change my old data? Google says custom channel groups can be applied to your reports retroactively, so yes, your history is relabelled in reports that use the group. Do I need to be an admin to create it? No, but you need at least the Editor role on the property. Sources Google Analytics Help: custom channel groups CitedBy free check, method note Keep reading Can ChatGPT read my website? What AI says about your brand The free crawler check What changed: log of AI-crawler and search changes ## GEO vs AEO vs SEO audit: what differs, what you need Source: https://citedby.eu/blog/geo-vs-aeo-vs-seo Guides / AI visibility basics GEO audit vs AEO audit vs SEO audit: what is different and which one you need By CitedBy · Page updated 22 October 2026. 2 sources · method Explanation written with AI from the official source cited on this page. Short answer An SEO audit checks whether you rank in search results. An AEO audit checks whether your pages give direct answers that assistants can quote. A GEO audit checks whether AI assistants name your brand in generated answers. The terms have no official definition, and the first two checks feed the third. On this page The short version What an SEO audit checks What an AEO audit checks What a GEO audit checks What Google says Which one a small shop needs first A five-step order of work Questions people ask Sources The short version None of the three terms is a standard. Each vendor draws the lines a little differently, so ask what a quoted audit actually tests before you compare prices. The grid below is how we use the words at CitedBy, and the sections after it explain each one. Our reading of the three terms. Other vendors group the checks differently. What an SEO audit checks An SEO audit asks whether search engines can crawl and index your pages and whether those pages deserve to rank for the searches your buyers make. Typical items are crawl access, page speed, titles, internal links, structured data and backlinks. The output is a list of fixes and a set of ranking positions. What an AEO audit checks AEO, answer engine optimization, asks whether a page answers a question so directly that a system can lift the answer. Typical items are a short answer under a question heading, FAQ blocks, tables and clear facts such as price, shipping time and returns. It overlaps with SEO, since a page that answers clearly also reads well to a search engine. What a GEO audit checks GEO, generative engine optimization, asks whether an assistant such as ChatGPT, Gemini or Perplexity names your brand when someone asks for a recommendation, and whether what it says is true. The check is run by asking real buyer questions, several times, and counting who gets named, who gets linked and which facts are wrong. A page-level checklist cannot answer that, because the result depends on what the assistant finds across the web. What Google says Google's documentation on AI features in Search states that there are no additional requirements to appear in AI Overviews or AI Mode and no other special optimizations necessary. It adds that the best practices for SEO remain relevant, that a page must be indexed and eligible to show with a snippet, and that you do not need new machine-readable files, AI text files or markup ( Google Search Central: AI features ). That applies to Google's own features. ChatGPT uses its own crawlers, and OpenAI lists three with different jobs: OAI-SearchBot for ChatGPT's search features, GPTBot for model training, and ChatGPT-User for actions a person starts ( OpenAI crawler documentation ). So the honest answer to "is it just SEO?" is: the base is shared, and the part that is new is measuring whether assistants name you. Which one a small shop needs first Your situation Start with Why Pages are not indexed, or the site is slow or empty without JavaScript SEO basics Nothing else works if crawlers cannot read the page You rank, but pages bury price, shipping and returns AEO fixes Direct facts are what an assistant can quote You rank and answer well, but buyers ask an assistant and rivals get named GEO audit This is the question only asking the assistant can answer You are not sure An audit, then 20 prompts by hand The audit shows crawler access on every page, the prompts show what is said A five-step order of work Run the free check in the box above and fix any crawler that is blocked or any page that is empty to a crawler. Make sure price, shipping, returns and a short description sit in the HTML text, not only in images or scripts. Add a short answer and a FAQ to your top three pages, written in your buyers' own words. Ask an assistant 20 buyer questions twice each and log who gets named ( the 20-prompt method ). Fix the sources behind any wrong fact, then repeat the prompts a few weeks later. Questions people ask Is GEO just SEO with a new name? Partly. The technical base is shared, and Google says SEO best practices remain relevant for its AI features. What GEO adds is measuring whether assistants name your brand, which a ranking report does not show. What is the difference between AEO and GEO? AEO is about making a page easy to quote as an answer. GEO is about whether an assistant mentions your brand in what it generates. The first is a property of your pages, the second is an outcome you measure by asking. Do I need an llms.txt file for any of them? Google says you do not need new machine-readable files or AI text files to appear in its AI features. We cover the evidence for other assistants in a separate guide on llms.txt. Which audit should I buy first? Fix crawl access and basic facts first, since you can check those yourself. Buy a GEO audit when you want to know who gets named when buyers ask, and ask the seller what prompts it runs and how often. Sources Google Search Central: AI features and your website OpenAI: crawlers and user agents CitedBy site audit Keep reading Can ChatGPT read my website? Check what AI says about your brand A sample report What changed: log of AI-crawler and search changes Profound vs Peec vs Otterly compared ## How to create an llms.txt file: 6 steps and a template (2026) Source: https://citedby.eu/blog/how-to-create-an-llms-txt-file Guides / AI visibility basics How to create an llms.txt file, step by step By CitedBy · Page updated 24 October 2026. method Explanation written with AI from the official source cited on this page. Short answer To create an llms.txt file, write a plain-text Markdown file with a # title, a short > summary and ## sections that list your key pages as [name](url): note links, save it as llms.txt and upload it to your site root so it loads at yourshop.com/llms.txt. Then run the free checker to confirm it loads and its links work. On this page What llms.txt is The format, in order Template to copy How to create the file Mistakes the checker finds Check the crawlers too Questions people ask Sources What llms.txt is llms.txt is a proposal published at llmstxt.org : a Markdown file at the root of your site that gives language-model assistants a short map of the pages that matter most. It is not a standard that any search engine or assistant has committed to read. Does llms.txt work? covers what the studies found. The file is cheap to write, and writing it makes you decide which of your pages answer buyers' questions. The format, in order The llmstxt.org specification lists these parts in this order. Only the first is required. A # title with the name of the site or project. A > blockquote with a short summary. Optional paragraphs or lists with more detail, but no headings. Zero or more ## sections, each holding a list of links. By convention, a section named Optional for links an assistant can skip when it needs shorter context. Each list item is a Markdown link, then optionally a colon and a note: - [Link title](https://link_url): what this page answers. Template to copy Replace the example text with your own. Keep the shape. # Example Shop > Example Shop sells handmade leather bags and ships across the EU. ## Products - [Leather bags](https://example.com/bags): all models with prices and sizes - [Care guide](https://example.com/care): how to clean and store a bag ## Policies - [Shipping and returns](https://example.com/shipping): delivery times and the 30-day return rule ## Optional - [About us](https://example.com/about) How to create the file Open a plain-text editor and type the title line: # Your shop name. Keep it to the name buyers know you by. Add one > line under it that says what you sell and where, in one sentence. Use facts a page of yours also states. List your most useful pages under ## headings such as Products, Help or Policies. Use full URLs, one link per line, and add a short note after a colon saying what the page answers. Short on time? The free llms.txt generator writes a first draft from your own pages' titles and descriptions for you to edit. Put pages that are nice to have, such as About or press, under a section named Optional. Save the file as llms.txt (plain text, UTF-8) and upload it to the root of your site so that https://yourshop.com/llms.txt loads. A file in a subfolder does not count, and the address must answer with text, not a login or an HTML error page. Run the free llms.txt checker on your domain. It reports whether the file loads, follows the format, links to pages that load, and agrees with your robots.txt and sitemap. Mistakes the checker finds The title is missing or is not the first line. A list item is not in [name](url) form, for example a bare URL. A listed page returns an error, or robots.txt keeps AI crawlers away from it. A page you list but block sends mixed signals. A listed page is not in your sitemap. The address returns an HTML page instead of a text file. Check the crawlers too A perfect llms.txt does nothing for a site whose firewall refuses AI crawlers. Run the AI-crawler access checker next: it tests robots.txt and a live request for eight crawlers. Questions people ask Where does llms.txt go? At the root of the domain: https://yourshop.com/llms.txt. Not in a subfolder. Do I have to list every page? No. List the pages that best answer buyers' questions. A short, accurate file is better than a long list of everything. Do ChatGPT, Gemini or Perplexity read llms.txt? No major assistant has confirmed that it does. Treat the file as a cheap addition, not as a way to get cited. What is llms-full.txt? An optional second file some sites publish with the full text of key pages in one document. The checker only reports whether it exists. How do I know the file is valid? Run the free llms.txt checker on your domain. Sources llmstxt.org: the /llms.txt file proposal Find what stops AI assistants reading your shop The audit crawls your site up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. Audit my site Audit from €9, paid once. 14-day money-back guarantee. Keep reading llms.txt generator (free) llms.txt checker (free) Does llms.txt work? AI-crawler access checker (free) Can ChatGPT read my website? Findable check: can search engines find your pages? (free) ## Is GPTBot blocked on your site? 4 ways to check (2026) Source: https://citedby.eu/blog/is-gptbot-blocked Guides / Crawlability Is GPTBot blocked on my site? Four places to check, with a test for each By CitedBy · Page updated 23 October 2026. 4 sources · method Explanation written with AI from the official source cited on this page. Short answer GPTBot is blocked on your site if robots.txt has a Disallow rule that applies to it, or if a CDN, firewall or server rule refuses its requests. Open yourshop.com/robots.txt, run an audit, then compare two requests to your server. Blocking GPTBot does not by itself block ChatGPT search. On this page Four places GPTBot can be stopped How to check, step by step The two-request test Cloudflare: where the setting lives When the check says your site refused the request Should you unblock GPTBot? Questions people ask Sources Four places GPTBot can be stopped GPTBot is the crawler OpenAI says it uses to collect content that may be used to train its models. A request from it can be turned away at four layers, and each one needs its own test. The free check in the box above covers the first. The other three answer with an error code even when robots.txt says yes. Layers 2 to 4 are invisible in robots.txt, so check them separately. How to check, step by step Open yourshop.com/robots.txt in a browser. Look for a group that names GPTBot. If there is none, GPTBot follows the User-agent: * group, so a blanket Disallow: / under it blocks GPTBot too ( RFC 9309 ). Run the free check in the box above. It reads robots.txt for GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended and Bingbot and names the ones that are blocked. Send two requests to your homepage and compare the status codes: one normal, one that carries the GPTBot user-agent string. Commands below. Open your CDN account and look at its AI crawler settings. For Cloudflare, the steps are below. Open your security plugin or firewall rule list and search for GPTBot, bot, AI and crawler. Search your server logs for GPTBot. A run of 403 responses to it is the sign of a block that robots.txt does not show. The two-request test Replace yourshop.com with your address. The first command sends a normal request, the second sends the GPTBot user-agent string that OpenAI publishes. Each prints only the status code. curl -s -o /dev/null -w "%{http_code}\n" https://yourshop.com/ curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://yourshop.com/ A 403 on the second request alone means a rule targets the crawler's name. Read the result with one limit in mind. This test comes from your own computer. If your host also judges crawlers by IP address, the real GPTBot can get a different answer from the one you saw. OpenAI publishes the IP ranges for GPTBot at openai.com/gptbot.json, so your logs can show whether requests from those ranges got through ( OpenAI crawler documentation ). Cloudflare: where the setting lives Cloudflare's documentation describes the steps: open your domain, go to AI Crawl Control, open the Security tab, then the Crawlers tab. The Actions column lets you Allow or Block each crawler. Choosing Block creates or updates a WAF custom rule, and you can still enforce robots.txt when you choose Allow ( Cloudflare: manage AI crawlers ). Cloudflare's defaults are also changing. Help Net Security reports that from 15 September 2026 new domains block Training and Agent crawlers by default on pages that show ads, while Search crawlers stay allowed ( Help Net Security ). Check your own dashboard rather than assuming which default your domain got. When the check says your site refused the request A plain request from our check can itself be refused. When the answer is HTTP 403, 429 or 503, the check says so and says it cannot tell from outside whether AI crawlers are refused as well. A 403 usually means a rule blocked the request. A 429 or 503 can also mean rate limiting or an outage, which affects every visitor. Fix the cause in your CDN, firewall or hosting panel, then run the check again. Should you unblock GPTBot? That is your decision, and blocking it is a legitimate choice. OpenAI lists GPTBot, OAI-SearchBot and ChatGPT-User as separate crawlers with separate jobs. GPTBot is for training data. OAI-SearchBot surfaces sites in ChatGPT's search features. ChatGPT-User fetches pages when a person asks about them. Your goal What to do Appear in ChatGPT search results, keep content out of training Allow OAI-SearchBot, disallow GPTBot Appear in ChatGPT search results and allow training Allow both Stay out of both Disallow both and check your logs for ChatGPT-User A change to robots.txt can take about 24 hours to reach OAI-SearchBot, according to OpenAI. Questions people ask Does blocking GPTBot remove my shop from ChatGPT? Not by itself. OpenAI describes OAI-SearchBot as the crawler for ChatGPT's search features and GPTBot as the training crawler, each set separately in robots.txt. Why does robots.txt look fine but GPTBot still gets a 403? robots.txt is a request to the crawler. A CDN, firewall or server rule can refuse the request anyway, so check those layers with the two-request test and your logs. Does ChatGPT-User obey robots.txt? OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts those requests. Will unblocking GPTBot get my shop recommended? No. Being readable is the entry ticket. Whether an assistant names your shop depends on what it finds about your category, which the CitedBy audit measures with real buyer questions. Sources OpenAI: crawlers and user agents, IP ranges Cloudflare: manage AI crawlers Help Net Security: Cloudflare changes AI crawler access rules, 2 July 2026 IETF RFC 9309: Robots Exclusion Protocol CitedBy site audit Keep reading AI crawlers: what each one does and how to allow or block it AI-crawler access checker (free) Can ChatGPT read my website? The CitedBy site audit A sample report Findable check: can search engines find your pages? (free) What changed: log of AI-crawler and search changes ## Is my website cited by ChatGPT? How to check it Source: https://citedby.eu/blog/is-my-website-cited-by-chatgpt Guides / Brand checks Is my website cited by ChatGPT? How to check it, step by step By CitedBy · Page updated 4 November 2026. method Explanation written with AI from the official source cited on this page. Short answer To find out if ChatGPT cites your website, run four checks: its crawlers can fetch your pages, Bing holds them in its index, ChatGPT names or links you when you ask the questions your buyers ask, and chatgpt.com shows up as a referrer in your analytics. A failure at step one or two explains most silence. On this page The four checks What stops sites at step 1 What stops sites at step 2 What to do with the result The faster way Questions people ask Sources The four checks Reach. Open your robots.txt and look for rules on GPTBot, OAI-SearchBot and ChatGPT-User, then fetch your homepage without JavaScript and count the words. The AI crawler checker does both for free. Index. ChatGPT search draws on search-engine results from partners, and Bing is the one most often named, so check that Bing and Google hold your pages: add your site to Bing Webmaster Tools and Google Search Console, submit the sitemap, and run URL Inspection on your homepage. Answers. Ask ChatGPT, logged out and in a new chat, the questions your buyers ask ("where can I buy [what you sell] in [country]?"). Write down whether it names you, links you and gets your facts right. Use the 20-prompt list. Visits. In GA4 or your server logs, filter referrers for chatgpt.com. A cited page sends clicks; a mention without a link may not. What stops sites at step 1 We fetched 300 sites that launched on Show HN (15 to 20 October 2026) and 70 built with Lovable, as CitedBot, with robots.txt obeyed and no JavaScript. The launch crawl found: Check Show HN Lovable Homepage under 100 words of text in the raw HTML 16% (47/293) 74% (49/66) robots.txt blocks at least one of five AI crawlers 4% (12/294) 0% (0/67) Homepage turned away a live request from one of those crawlers 11% (33/293) 0% (0/66) So robots.txt was the smaller factor. Among the Show HN sites, an empty HTML shell and a firewall that refuses AI user agents turned up more often. These numbers say nothing about whether those sites are cited; we did not measure that. The Lovable column is what CitedBot got. On Lovable's hosting, apps created before 13 May 2026 (22 June on Enterprise plans) send a pre-rendered copy to crawlers Lovable verifies, its docs say. These include the AI search crawlers of ChatGPT, Perplexity, Claude and Gemini. Newer apps send full HTML to every visitor, and any client that asks for Markdown gets a Markdown copy ( Lovable docs, read 4 November 2026). We did not measure what those crawlers received. So for an app on Lovable's hosting, this column is no evidence that ChatGPT's search crawler sees an empty page. What stops sites at step 2 A page that no index holds cannot be found by a search-backed answer. On 22 October 2026 Bing's own URL Inspection reported the home pages of three of our sites as "Discovered but not crawled". Discovered means Bing knows the URL and has not fetched it yet. Submitting the sitemap and the IndexNow protocol on every publish shortens the wait; it does not set a date. What to do with the result Step 1 fails: fix robots.txt rules or the firewall first, then make the homepage send its text in the HTML. Step 2 fails: submit the sitemap, request indexing and ping IndexNow, then check again in a week. Step 3 shows rivals, not you: open the pages ChatGPT linked for them and note what they hold that yours lacks. Step 4 shows nothing: one week is too short to judge; log the date and repeat monthly. The faster way The audit crawls your site up to your plan's page limit, shows what AI crawlers cannot reach or read, and gives the exact fix for each issue. It starts at €9. Prefer to check only access first? Run the free check. Questions people ask Can I see every time ChatGPT cites my site? No. OpenAI does not publish a citation report for site owners. Your server logs show its crawlers' requests and visits from chatgpt.com, and asking the questions by hand shows the rest. Does blocking GPTBot stop ChatGPT citing me? Not by itself. GPTBot is the training crawler; OAI-SearchBot is the one used for search, and ChatGPT-User fetches pages when a person asks. Check each rule on its own. How long until a new page is cited? We cannot give a number. It must first be crawled and indexed, and answers change from run to run. Sources CitedBy launch crawl 2026-10: method, limits and CSV OpenAI: crawlers overview (GPTBot, OAI-SearchBot, ChatGPT-User) About CitedBy: how we count Keep reading AI crawler checker Is my site findable by Google, Bing and ChatGPT? Launch crawl: 370 sites 20 prompts to ask ChatGPT about your brand ## How a template flood cost us 433 of 464 Google pages Source: https://citedby.eu/blog/template-flood-young-site-case Guides / Our own data How a template flood cost a young site of ours 433 of 464 Google pages By CitedBy · Page updated 4 November 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer A young site of ours went from 6 to 1,806 sitemap URLs in five days, then fell from 612 to 44 Google impressions a day. 433 of the 464 pages that had impressions on 23 to 25 October had none on 26 and 27 October. We wrote three sitemap checks from it. Sitemap URLs 6 on 17 Oct, 1,806 on 22 Oct Impressions a day 612 on 25 Oct, 44 on 26 Oct Pages with impressions 433 of 464 had none on 26-27 Oct Our grade A 96, then C- 72 On this page What we did What Search Console showed If your site is in this spot this week What we do not know The three checks we wrote from it Our own grades before and after What we do now Questions people ask Sources What we did The site is a young one of ours: its first release was on 14 October 2026. Between 17 and 22 October we put out one template of pages after another. Our own count of the sitemap was 6 URLs on 17 October and 1,806 on 22 October. It reached 2,183 on 27 October. Our audit of 27 October read what those pages looked like: 1,463 of the 2,183 URLs used one template, the question pages. Their median length was 412 words, and in the pages we sampled 44% to 58% of a page's text matched a sibling page. Every page carried the same product block. 106 of 180 sampled meta descriptions were the same sentence with a different number. We also changed the site a lot while Google was still reading it. We rewrote titles and descriptions on those pages after launch and released several times a day. The sitemap's last-modified date moved with each release: 869 of the 1,439 sitemap URLs the 28 October crawl read carry the date 2026-10-27 (the next busiest day has 192), and we pinged IndexNow for 5,563 URLs in seven days. What Search Console showed For a week the curve looked like success. Impressions went from 2 a day on 17 October to 612 on 25 October, and the number of pages with at least one impression grew with it. Then it stopped. Source: Search Console, Search results, read 28 Oct 2026; sitemap URLs are our own daily count. The pages that had at least one impression each day tell the same story: Day (2026) Pages with impressions 20 Oct 31 22 Oct 119 23 Oct 224 24 Oct 219 25 Oct 259 26 Oct 30 27 Oct 20 Search Console, Search results by page, read 28 October 2026. The daily impressions and sitemap counts in the chart are in the case CSV. Counting distinct pages gives the figure in the title. 464 different pages had at least one impression on 23, 24 or 25 October. On 26 and 27 October, 31 of them still had one and 433 (93%) had none. Adding 28 October, whose data was not final when we first read it, 35 still had one and 429 had none (Search Console final data, read 1 November 2026 at 23:34 UTC). Most of those pages did not slip down the results; they stopped appearing at all. Corrected on 2 November 2026: this count and the short answer said 26 to 28 October. The 433 holds for 26 and 27 October. The index report moved later than the impressions. We inspected the same 254 URLs, taken across every page group, through the URL Inspection API twice: Source: URL Inspection API, the same 254 URLs read on 26 Oct 01:45 UTC and 27 Oct 15:24 UTC. On 26 October, 237 of the 254 were indexed and 1 was "Crawled, currently not indexed". On 27 October, 173 were indexed and 66 were "Crawled, currently not indexed"; 16 and then 15 were unknown to Google. On 27 October the Manual actions page said "No issues detected". URL Inspection showed a successful fetch, indexing allowed and Google's canonical equal to ours. Googlebot, read from verified Google addresses in our logs, got HTTP 200 throughout the logs we have for 19 to 27 October, apart from one 504 on 22 October. Most of 24 October is missing from those logs. We found no technical fault. If your site is in this spot this week What we did. Steps 1 to 3 told us what happened; whether step 5 brings the pages back, we do not know yet: Open Manual actions in Search Console. If it lists one, that is a different problem from ours. Inspect a fixed sample of your URLs now and again a day later, and count how many move to "Crawled, currently not indexed". Count how many URLs your sitemap gained in the last 30 days, and how many share one URL pattern. Stop releasing changes to titles, descriptions and last-modified dates on those pages. Take the thin template pages out of the sitemap and noindex them. Add no URLs, and read Search Console once a week. What we do not know We cannot prove what caused it. The most likely cause is a site-wide quality reassessment of a 13-day-old site with about 1,800 template pages that repeated much of each other's text. Google's Search Status Dashboard lists no ranking update for 26 or 27 October; the last one before it, the September 2026 spam update, ended on 8 October (read 4 November 2026). Other site owners report the same shape: on r/TechSEO one reports Google traffic falling by more than 95% overnight on 7 October, inside that spam update by the poster's dates, on a site that had been getting 8,000 to 10,000 impressions a day, and is not sure of the cause either. We changed pace, wording and dates at the same time, so we cannot say which one counted most. We have no run of the same site without the flood to compare against. The three checks we wrote from it After the drop we wrote three sitemap checks into our site audit. We have tested them on this one site so far. On the 28 October crawl of that site all three fired: template_flood: "Most of the sitemap is one template" What it reads: URL patterns with at least 100 sitemap URLs and 8% of the sitemap, where the pages we read share a median 25% or more of their text with their nearest sibling, at least three times what they share with other pages. It warns from half the sitemap. On that site, 28 October: 845 of 1,439 sitemap URLs (59%) are one template under a single URL prefix (the 1,463 of 27 October, after the cut described below). The 845 pages we read share a median 39% of their text with the nearest page of the pattern, against 7% with other pages. sitemap_publish_spike: "Many sitemap URLs are dated the same day" What it reads: one day holding at least 150 dated URLs and 30% of the dated sitemap. It warns from 500 URLs and half. On that site, 28 October: 869 of 1,439 dated URLs (60%) carry 2026-10-27. index_likely_not: "Many sitemap pages will probably not be indexed" What it reads: an estimate from robots.txt blocks, noindex, canonical elsewhere, errors, near-duplicate pages and pages with almost no text, read from the pages we fetched per template and applied to the rest. It warns from 25% of the sitemap. On that site, 28 October: an estimated 441 of 1,439 sitemap URLs (31%) will probably not be indexed, from 1,192 pages read, all 441 near-duplicates of a sibling. Each check lists the page examples and the fix. We wrote these checks after the drop, with that site as the test case, so it is the one site we describe them on. The crawl of 26 October, before they existed, did not warn us. Our own grades before and after Crawl of that site Grade 26 Oct, before the three checks A 96 27 Oct, three crawls the same day A+ 100, then A 94, then C- 72 28 Oct, with the checks C- 72 The grades are in the case CSV. The site did not get worse between 26 and 28 October. The grade moved because we added checks. The 28 October report holds the grade at C- because the crawl found three signs that Google will not index much of the site, and it states that the score without that limit is 96 out of 100. The cap exists because a site can look clean page by page and still be one Google declines to index. What we do now On 28 October we cut the sitemap from 2,183 to 1,439 URLs and noindexed pages (the 28 October crawl lists 835 pages that cannot be indexed, mostly because of noindex). We have added or removed no URL since, and we plan to leave the site's URLs alone for 12 to 16 weeks from 28 October, rewriting only pages it already has. We also add none until our drop alarm for the site has been quiet for 7 days. Since 3 November we have been rewriting the main text of a few existing pages in place, on a plan that runs to 10 November. Titles, slugs and canonicals stay the same. A change after 3 November therefore cannot be put down to the cut alone. So far the site's Google impressions have not come back. In the five days after the cut, 29 October to 2 November, the site had 3 Google impressions in total (2, 0, 0, 1 and 0 a day), against 612 on 25 October alone (Search Console, read 3 November 2026 at 23:14 UTC; 2 November was not yet final; the days are in the case CSV ). We will add the next read to this page. On our own sites we now work to these rules. We have not yet seen them bring a site back: Before a site adds or removes pages, it has a written page plan that caps how many new pages it may add a day and says whether a new template first goes live on 1 to 3 pages. A template page goes live only with at least 60% of its text its own, and its own title and description. A page's last-modified date moves only when its main text does, and IndexNow is pinged for changed pages only. No title, slug or canonical change in a page's first 28 days. For 3 to 9 November, three of our four page plans, this site's among them, add no new pages and the fourth allows at most one hand-written page a day. Corrected on 2 November 2026: rule 1 said at most 50 new pages a week. On 30 October that became 5 a day and 35 in 7 days. The Googlebot line earlier in this post said HTTP 200 on every day from 19 to 27 October; most of 24 October is missing from our logs, so it now covers only the logs we have. Updated on 4 November 2026: until 3 November one pace rule held on every site. It allowed 5 new pages a day and 35 in 7 days (15 for a domain under 90 days old), and the next batch only once 70% of the last was indexed after 14 days. Since 3 November each site's own plan sets these limits, so rule 1 now says that and the batch rule is no longer on the list. This section also said we would leave this site alone for 12 to 16 weeks; it is the site's URLs we leave alone, and since 3 November we rewrite pages it already has. The 24 October log line no longer gives a number of hours, because our two sources disagree on how much of that day is missing. Questions people ask Why did my Google impressions drop overnight? A drop of more than 90% in a day, with pages still in the index and no manual action, is what we saw after publishing many pages from one template. Check Search Console's Manual actions first, then how many URLs your sitemap gained in the last month and how much text each page shares with its sibling. Does "Crawled, currently not indexed" mean Google found a fault? No. It means Google fetched the page and chose not to index it. On that site the count of such pages in our fixed sample went from 1 to 66 of 254 between two reads 38 hours apart. Sources Search Console data for the site, read 28 Oct and 3 Nov 2026: impressions, clicks and sitemap counts by day (CSV) Google Search Central: Manual actions report Google Search Status Dashboard: ranking updates, read 4 Nov 2026 r/TechSEO 1xccp1n: Google traffic dropped 95% overnight Keep reading The case as a CSV: dates, impressions, sitemap URLs, grades A sample report Findable check: can search engines find your pages? (free) ## Lovable, Bolt or React site empty to AI? Check and fix Source: https://citedby.eu/blog/vibecoded-site-empty-to-ai Guides / Crawlability My Lovable, Bolt or React site looks empty to AI. How to check it and fix it By CitedBy · Page updated 26 October 2026. 3 sources · method Explanation written with AI from the official source cited on this page. Short answer AI crawlers read the HTML your server sends and, according to Vercel's analysis, do not run JavaScript. A site built in the browser can look empty to them even when it looks perfect to you. Test it by viewing the page source, then switch to server rendering or prerendering if your text is missing. On this page Why a site that looks fine can be empty to AI The 30-second test What Lovable says Fixes by setup Two other things to check at the same time Questions people ask Sources Why a site that looks fine can be empty to AI A site made with an AI app builder is often a single-page app. The server sends a near-empty HTML shell, and JavaScript builds the headline, price and buttons in your browser. A visitor never notices. A crawler that does not run the script receives the shell and nothing else. Vercel analysed AI crawler traffic across its network and reported that none of the major AI crawlers it looked at render JavaScript. It names OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Perplexity (PerplexityBot), Meta and ByteDance. It also reports that ChatGPT's and Claude's crawlers fetch JavaScript files (11.5% and 23.84% of their requests) without executing them, while Gemini uses Googlebot's infrastructure and can render JavaScript ( Vercel: the rise of the AI crawler ). That is one company's measurement from its own network, and crawler behaviour can change, so test your own site instead of assuming. What a crawler without JavaScript receives, in both setups. The 30-second test Open your homepage in a browser, right click and choose View page source (not Inspect, which shows the page after scripts ran). Search the source for your main headline and one price or product name. If you find them in the text, crawlers can read them. If you find only a
and script tags, the page is empty to a crawler that skips JavaScript. Run the free check in the box above. It fetches your page with a plain request, without running scripts, and counts the readable words in the HTML. For a command-line version, curl -s https://yourshop.com/ | head -c 3000 prints the start of what a crawler gets. What Lovable says Lovable's FAQ says new apps created from 13 May 2026 (22 June 2026 for Enterprise workspaces) use TanStack Start, which renders pages on the server, and that older apps use React with Vite and render in the browser. For those older apps it says Lovable serves pre-rendered copies of published pages to verified search engines, social preview bots and AI crawlers ( Lovable FAQ ). So a Lovable site may be readable even if it is client-rendered, but that depends on how and where it is published. A custom setup or a copy exported to another host does not inherit it. Check the source on your live address, not on the builder's preview. We found no equivalent statement from Bolt or other builders, so this article covers only the builders whose statements it cites. Run the test on each site. Fixes by setup Your setup Fix Check afterwards New Lovable app (TanStack Start) Nothing to switch, rendering is on the server View source on the live address Older Lovable app (React + Vite) Confirm the live address returns the text, and read Lovable's rendering notes for options View source, and the audit React + Vite you host yourself Move critical pages to server rendering or prerender them at build time View source on each key page Next.js Pages and layouts are Server Components by default, which render on the server ( Next.js docs ). Keep price and description out of client-only components View source Other builder Ask the vendor how published pages are rendered, then test View source Whatever you choose, the pages that matter are the ones a buyer would ask about: home, category, product, shipping and returns. Put price, availability and a short description in the server-sent HTML and keep scripts for interaction only. Two other things to check at the same time Product data: the price and name should also appear as Product JSON-LD in the server HTML, not injected later. Blocks: a site can be readable and still be refused. See Is GPTBot blocked on my site?. Questions people ask Does ChatGPT run JavaScript on my site? Vercel's analysis found that none of the major AI crawlers, including OpenAI's, render JavaScript. Test your own page with View page source, since behaviour can change. How do I know if my site is client-side rendered? View the page source. If your headline and price are not in it, and you see an empty root div with script tags, the page is built in the browser. Is a Lovable site always empty to AI? No. Lovable says new apps render on the server and that older apps get pre-rendered copies served to verified crawlers. Check the live address to be sure. Does Google see my client-rendered site? Vercel reports Gemini uses Googlebot's infrastructure and can render JavaScript. Other assistants may still read nothing, so server rendering helps with all of them. Sources Vercel: the rise of the AI crawler Lovable FAQ: rendering and SEO Next.js: Server and Client Components CitedBy site audit Keep reading Can ChatGPT read my website? Is GPTBot blocked on my site? A sample report Findable check: can search engines find your pages? (free) What changed: log of AI-crawler and search changes