Google-Extended
Google-Extended is a robots.txt token that controls whether content Google crawls can be used to train and ground its generative AI products — Gemini models in Gemini Apps, the Vertex AI API for Gemini, and grounding in both. Google states plainly that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It is not a crawler; Googlebot does the fetching and this token governs the downstream use.
It is a use control, not a crawler
This is the part that trips people up. There is no bot called Google-Extended fetching your pages. Googlebot crawls as it always has; the token tells Google whether the resulting content may be used for generative AI training and grounding in Gemini Apps and the Vertex AI API for Gemini.
So disallowing it does not reduce crawl load, does not change your indexing, and does not remove you from anything in Search.
What it does not cover: AI Overviews and AI Mode
The single most consequential misunderstanding in this area. Google-Extended does not opt you out of AI Overviews or AI Mode, because those are surfaces within Google Search rather than separate products. Google's guidance on AI features in Search is explicit that appearing in them requires only that a page be indexed and eligible to appear in Search with a snippet.
The controls that do affect AI features are the ordinary preview controls: nosnippet, data-nosnippet, max-snippet, and noindex. And they are blunt instruments — nosnippet suppresses your snippet across Search, not only in AI features. Publishers looking for a surgical opt-out of AI Overviews alone do not have one.
Deciding for a product catalog
Product specifications are facts. Facts are not copyrightable and are widely republished across distributors, manufacturer sites, and data pools, so the case for withholding them from training is weaker than it is for a news publisher.
The stronger case sits around genuinely proprietary content: curated cross-reference databases, application engineering guides, fitment data you built rather than received. If that is a real asset, Google-Extended: Disallow is a coherent position, and it costs you nothing in Search.
Most distributors allow it, on the reasoning that being present in the model's own knowledge of the category is worth more than protecting spec tables that everyone else publishes anyway.
Syntax
User-agent: Google-Extended
Disallow: /
Applies at path level like any other robots.txt directive, so you can withhold /guides/ and /cross-reference/ while allowing product pages. Note also Google-CloudVertexBot, a distinct agent that crawls sites when a site owner explicitly requests it for building Vertex AI Agents — different mechanism, different purpose, not something that visits uninvited.
Frequently asked questions
Does Google-Extended block AI Overviews?
No. AI Overviews and AI Mode are features within Google Search, and Google states that Google-Extended does not affect inclusion in Search. Appearing in AI features requires only being indexed and snippet-eligible. The controls that affect them are nosnippet, data-nosnippet, max-snippet and noindex.
Does disallowing Google-Extended hurt my rankings?
Google states directly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It governs whether crawled content can be used for training and grounding in Gemini Apps and the Vertex AI API for Gemini.
Is Google-Extended a crawler?
No. It is a robots.txt token that controls downstream use of content Googlebot already crawled. No separate bot fetches your pages under that name, so disallowing it does not reduce crawl traffic.
Should a distributor disallow Google-Extended?
It depends on whether you hold proprietary content worth withholding. Specifications are facts published in many places, so the protective value is low. Curated cross-reference and application-engineering content is a stronger candidate, and you can disallow it by path rather than sitewide.