Checks / AI crawler access
robots.txt blocks Google-Extended
Google-Extended controls whether Google uses your content for Gemini training and grounding. Blocking it is a choice; it does not affect Google Search.
robots_google_extended · AI crawler access · Notice
What triggers it
robots.txt does not let Google-Extended fetch your home page (a Disallow rule in its own group, or in the * group when no group names it), and it leaves no other path open for it.
What to change
Allow Google-Extended if you want your content used by Gemini.
Example
Lines that allow Google-Extended in robots.txt:
User-agent: Google-Extended
Allow: /How to confirm the fix
- Open https://example.com/robots.txt: no group that names Google-Extended (or the * group, if none does) disallows the home page
- Run a new crawl: the AI crawler table lists Google-Extended as allowed
About Google-Extended
Google-Extended is a robots.txt product token that lets publishers manage whether content Google crawls may be used to train future Gemini models. It has no HTTP user agent of its own, so it never shows in your logs. Google says it does not affect inclusion in Search or ranking.
Adding `Disallow: /` under Google-Extended tells Google it may not use the content for training future Gemini models behind Gemini Apps and the Vertex AI API for Gemini. Google also says it does not impact inclusion in Google Search and is not a ranking signal. Our audit reads this token in robots.txt as one of its checks; it is the only signal there is. Google's page also names grounding, meaning content from the Google Search index given to the model at prompt time, as something this token manages.
Job: Control token (no requests of its own). Follows robots.txt: Read from robots.txt only; there is no separate user agent.
Google-Extended: operator, user-agent string and address list
How common it is
On a sample, the study measured robots.txt blocking Google-Extended on the homepage's site: 9 of 294 Show HN sites (3%), and 0 of 67 Lovable sites (0%).
Fetched 20 and 21 Oct 2026 as CitedBot, robots.txt obeyed, no JavaScript. Study: Launch crawl 2026-10 (English).
About this check
| Check ID | robots_google_extended |
|---|---|
| Report section | AI crawler access |
| Severity | Notice |
| How it is decided | Measured: read directly from the page or the server's response. |
| Added | Before 19 Oct 2026 (versions 1 and 2), or a site-wide check with no version date yet |
| Status | Measured in every report |
| Report gives the code | Yes, one block for the site |
Settings by platform
For robots.txt, where the setting lives differs by platform. Our guides cover 13 of them: AI crawler settings by platform.
Related checks
- robots.txt blocks CCBot
- robots.txt blocks ClaudeBot
- robots.txt blocks GPTBot
- robots.txt keeps CCBot out of your home page and most of the site
See whether it applies to your site
Paste your address and we crawl up to 10 pages and show every issue we find, with its fix. No account, no card.
Every plan crawls up to its page limit and lists these checks with the fix: plans.
How an audit works: method