Does Blocking AI Crawlers Hurt Your Google Rankings?

The short version, in Google's own words

Google publishes a page specifically describing its crawlers and fetchers, and it draws the line as clearly as a document like this ever does. On Google-Extended, the page states: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

On Googlebot, the same page describes its purpose as crawling "to find information for building Google's search indexes, perform other product specific crawls, and for analysis." Two different jobs, stated plainly, by the company that runs both.

What Google-Extended actually is

Here's the part that surprises most people: Google-Extended isn't a separate bot that shows up at your website with its own name tag. Google's documentation is specific about this — "Google-Extended does not operate via a separate HTTP user agent string. Rather, crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."

In plain terms: the same crawl Googlebot already does can serve two purposes. Google-Extended is the instruction you give in your robots.txt file about whether that already-crawled content may also be used to train future Gemini models and to ground Gemini's answers. It's a permission setting layered on top of an existing visit, not a second visitor.

How to set them independently, in plain terms

Because Googlebot and Google-Extended are governed by separate robots.txt rules, you get to choose each one on its own. A business that wants to stay fully visible in Google Search results, but doesn't want its content feeding Gemini's training or its answers, can do exactly that: allow the Googlebot user agent as normal, and add a separate Disallow rule for the Google-Extended user agent.

The reverse is also true — you could theoretically block Googlebot (and vanish from Google Search entirely, which almost no business wants) while allowing Google-Extended, though in practice almost nobody has a reason to do that. The point is simply that these are two levers, not one, and pulling one doesn't move the other.

A concrete example: a family-run bakery

Picture a bakery owner who read online that "blocking AI bots hurts your SEO" and, out of fear, left everything wide open rather than risk her Google ranking — including crawlers she'd have preferred to keep out.

With the facts above, she has a cleaner choice: her robots.txt file can explicitly allow Googlebot (keeping her bakery fully searchable on Google, exactly as before) while explicitly disallowing Google-Extended (opting her menu descriptions and photos out of training future Gemini models) — without touching her Search visibility at all. The decision about Gemini training becomes a separate, deliberate choice instead of a side effect she's guessing about.

Why this confusion is so common, and how to avoid it going forward

The confusion is understandable: robots.txt has historically been almost exclusively a Search-ranking tool, so any new line added to it feels like it must be about ranking too. Google-Extended breaks that old pattern — it's the first mainstream robots.txt token that is explicitly, by Google's own statement, unrelated to ranking.

The safest habit going forward is simple: don't guess. When a new crawler or token shows up, check that company's own documentation for what it actually controls before changing anything — the same discipline this article is built on.

FAQ