AI crawlers / AI crawlers
Google-CloudVertexBot: what it does and what blocking it changes
What Google-CloudVertexBot is
Google-CloudVertexBot is a Google crawler listed among its common crawlers. Google says it performs crawls requested by site owners for building Vertex AI Agents. Google shows it as a user agent substring, not a full user agent string.
| Fact | Google-CloudVertexBot |
|---|---|
| Operator | |
| Job | Crawl requested by site owners for Vertex AI Agents |
| robots.txt token | Google-CloudVertexBot |
| Follows robots.txt | Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. |
| IP addresses | 317 ranges (developers.google.com/static/crawling/ipranges/common-crawlers.json, file dated 2026-10-23) |
| Vendor page read | 24 October 2026 |
What the vendor says
In the vendor's words (English original): “Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents”
Source: Google Search Central: Google's common crawlers, read on 24 October 2026.
What happens if you block Google-CloudVertexBot
Google says preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by site owners for building Vertex AI Agents. It says this has no effect on Google Search or other products.
The robots.txt lines
To allow Google-CloudVertexBot:
User-agent: Google-CloudVertexBot
Allow: /To block Google-CloudVertexBot:
User-agent: Google-CloudVertexBot
Disallow: /Paste a block into the robots.txt at the root of your host, for example https://example.com/robots.txt. A group for one token does not cover the operator's other crawlers.
The user-agent string the vendor shows:
Google-CloudVertexBotHow to tell a real request from a fake one
The file at developers.google.com/static/crawling/ipranges/common-crawlers.json held 317 ranges on 24 October 2026, file dated 2026-10-23.
Google says the common crawlers generally crawl from the ranges in common-crawlers.json, and that the reverse DNS name matches crawl-*-*-*-*.googlebot.com or geo-crawl-*-*-*-*.geo.googlebot.com.
Compare with Googlebot
Googlebot: Googlebot crawls for Google Search, while Google-CloudVertexBot only crawls when a site owner has requested a crawl for building Vertex AI Agents.
Your own site
The free check in the box above tests eight crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot and Googlebot). Google-CloudVertexBot is not one of them: read your robots.txt for a Google-CloudVertexBot group and look for the user agent in your server log. The full list of crawlers is on the AI crawler hub.
Questions people ask
Does Google-CloudVertexBot follow robots.txt?
Yes. Google says common crawlers always obey robots.txt rules when crawling automatically. Give Google-CloudVertexBot its own User-agent group in robots.txt: the operator's other crawlers have their own settings.
How do I know Google-CloudVertexBot is really blocked?
robots.txt is only one layer: a CDN or firewall rule can also turn a crawler away, and a crawler can ignore the file. Check your server log for the user agent, and test the address from outside your network.
Sources
- Google Search Central: Google's common crawlers
- Google: common-crawlers.json (read 2026-10-24, creationTime 2026-10-23)
- Google Search Central: Overview of Google crawlers and fetchers
See what AI answers about your shop
The audit crawls your site up to your plan's page limit, shows which AI crawlers it lets in and gives the exact fix for each issue. From €9, 14-day money-back guarantee.
Audit my siteAudit from €9, paid once. 14-day money-back guarantee.