"Google is ignoring robots.txt for AI" is half true, and the half that is wrong is the half that matters for SEO. The claim collapses three very different things into one. Googlebot — the crawler that indexes your site for Search and feeds AI Overviews — still respects robots.txt exactly as it always has. Google-Extended, the token Google introduced in 2023, still respects it too, and cleanly lets you opt your content out of Gemini training and grounding with zero effect on your search rankings. Neither of those is ignoring anything. What is genuinely ignoring robots.txt is a third, growing category: Google's user-triggered fetchers — the bots that fetch a page because a person asked a Google product to, like Google-GeminiNotebook (renamed from Google-NotebookLM in July 2026), Google-Agent, and Google Read Aloud. Google's own documentation states plainly that because the fetch was requested by a user, these fetchers generally ignore robots.txt, and there is no way to change that with a robots.txt rule, because robots.txt is a request, not an enforceable directive. This guide separates the three cleanly, explains why the user-triggered class ignores robots.txt by design, walks through the real SEO tension it creates — that you cannot opt out of AI Overviews without also delisting from Search, because both run on the crawler that does obey your rules — and covers how to actually control AI access when robots.txt does not, from server-level user-agent blocking to IP-range firewalls. It closes on the strategic conclusion the crawl-control debate keeps dancing around: the surfaces that scrape you now often return zero referral traffic, so the durable move is to stop depending on a crawler to carry your content and start publishing it directly onto every platform yourself.
"Google is ignoring robots.txt for AI" is one of those claims that is technically true, mostly false, and confusing in exactly the way that leads to bad decisions. The problem is that it collapses three completely different things Google does to your website into a single sentence. Two of them still respect robots.txt precisely as they always have. One of them ignores it by design — and Google says so openly in its own documentation. If you act on the headline without separating the three, you will either panic-block a crawler that was never the problem and quietly delist yourself from search, or shrug off a real change in how your content is being taken and reused. This guide splits them apart and then works through what each one means for your SEO and your distribution.
Here is the whole thing in one line before the detail: Googlebot obeys robots.txt, Google-Extended obeys robots.txt, and Google's user-triggered fetchers do not. The first two are the ones your SEO depends on. The third is the one people are actually talking about when they say Google is ignoring robots.txt for AI. Once you see which is which, the strategy becomes obvious — and it is not the strategy the crawl-blocking debate usually lands on.
Google does not have one bot. It has a crawler that indexes the web, a token that governs AI-training use, and a whole class of fetchers that grab a single page on demand. They behave differently, they are controlled by different mechanisms, and conflating them is the root of almost every wrong conclusion about robots.txt and AI.
Googlebot is the crawler that indexes your site for Google Search, and it still honors robots.txt exactly as it has for two decades. This matters more than it sounds, because AI Overviews — the AI-generated answers at the top of the results page — are built from the same index Googlebot creates. There is no separate "AI Overviews crawler" and no dedicated token to opt out of them. So when Googlebot obeys your robots.txt, it is obeying it for both classical search results and AI Overviews at once. That single fact is the source of the real SEO tension in this whole subject, and we come back to it below.
Google-Extended is not a crawler at all; it is a token you place in robots.txt, introduced in September 2023. Disallowing it tells Google not to use your content to train its generative models or to ground answers inside Gemini apps and Vertex AI. Google has been explicit that this has no effect on Google Search — your indexing and rankings are untouched, because the crawling is still done by Googlebot, and Google-Extended only governs the downstream use of what was crawled. It is the one clean, respected lever publishers have to separate "use my content to train AI" from "rank my content in search." It works because it is honored. It just does not control AI Overviews, and it does not touch the fetchers in the next section.
This is the category that earned the headline. User-triggered fetchers are bots that fetch a specific page because a person asked a Google product to — you paste a URL into Gemini Notebook and it goes and reads that page, or Google Read Aloud fetches an article to speak it. Google's crawler documentation states the behavior directly: because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The class includes Google-GeminiNotebook (renamed from Google-NotebookLM in July 2026, with the old token supported only until around August 2026), Google-Agent, Google Read Aloud, Feedfetcher, Google Pinpoint, and others. None of them consult your robots.txt before fetching, and no robots.txt rule will stop them. This is not a bug or a policy drift — it is the documented, intended design of the entire user-triggered class.
There are two reasons stacked on top of each other, and understanding both keeps you from expecting a fix that is never coming. The first is Google's framing: it treats a user-triggered fetch as a person visiting the page through a tool, not as an autonomous crawler indexing the web. robots.txt was written to govern automated crawling; in Google's reading, a fetch that a specific human just requested falls outside that scope, the same way robots.txt does not stop a person from opening your page in a browser. Whether you accept that framing or not, it is the stated rationale, and it is why the behavior is documented rather than hidden.
The second reason is more fundamental and it is worth internalizing: robots.txt has never been an enforceable directive. It is a voluntary request that well-behaved crawlers choose to honor. There is no technical mechanism that forces compliance — a bot that reads your robots.txt and ignores it is not bypassing a lock, because there was never a lock, only a sign. Googlebot honors the sign as a matter of long-standing policy. The user-triggered fetchers, by policy, do not. This is the same reason less scrupulous third-party AI crawlers can ignore robots.txt entirely, and the reason blocking AI crawlers is more complicated than adding a disallow line. If you want a rule with teeth, robots.txt is the wrong layer — you have to move down to the server or the network, which we cover shortly.
The reason this jumped from a footnote in crawler docs to a widely discussed issue was Google's rename of NotebookLM to Gemini Notebook in July 2026, which changed the fetcher's user agent from Google-NotebookLM to Google-GeminiNotebook. The fetcher itself had been documented as a user-triggered fetcher since October 2025, ignoring robots.txt the whole time, but the rename meant that any site owner who had been trying to track or block the old user-agent string would silently stop matching it — the legacy token is supported only until around August 2026. Overnight, monitoring rules that looked like they were working were quietly doing nothing.
The substantive concern underneath the naming churn is what the product does with the fetch. Gemini Notebook's source-discovery feature can pull a batch of articles per user query, summarize them inside the tool, and return no referral traffic to the originals — the reader gets the synthesized answer and never clicks through. That is the same zero-click dynamic gutting publisher referral traffic across AI discovery, arriving through a fetcher you cannot turn off with robots.txt. For the specific event and its mechanics, see the write-up of the Gemini Notebook crawler rename and the broader rebrand. The pattern it represents — on-demand AI fetchers that read your content, answer for you, and send nothing back — is the durable story, not the one product.
Here is the trap that catches publishers who react to the "Google ignores robots.txt for AI" headline by reaching for a disallow rule. The AI feature most likely to be eating your clicks is AI Overviews, not the user-triggered fetchers — and AI Overviews are the one thing you cannot cleanly block. Because they are generated from Googlebot's index, the only robots.txt move that keeps you out of AI Overviews is disallowing Googlebot itself, which also erases you from ordinary search results. There is no middle setting. Google-Extended does not help, because it governs training and grounding, not AI Overviews eligibility. So the crawler that actually respects your robots.txt is the one you most wish you could partially restrain, and you cannot, without paying the full price of leaving search.
That is the honest shape of the SEO impact, and it is close to the opposite of the panic the headline invites. The fetchers that ignore robots.txt (Gemini Notebook and its class) are on-demand and relatively low-volume. The feature that is measurably reducing organic clicks is AI Overviews, which runs on the crawler that does obey you — meaning your only lever there is a self-destruct button. Spending your energy trying to block the fetchers while ignoring the Overviews dynamic is optimizing the wrong variable. The productive response is not crawl control at all; it is accepting that your content will be read and summarized by machines you cannot gate, and changing where you place your bets accordingly.
If you have a genuine reason to keep a specific AI fetcher off a specific set of pages — gated premium content, a paywall, licensed material — you have to enforce it below robots.txt, because robots.txt is a request the fetcher declines. There are three practical layers, in ascending order of reliability.
The first is user-agent blocking at the server: match the fetcher's user-agent string in your server config, .htaccess, or CDN rules and return a 403. It is the simplest to set up and the easiest to evade, because user agents can be omitted or spoofed, and because a rename like the NotebookLM-to-Gemini-Notebook switch silently breaks any rule pinned to the old string. The second, and sturdier, is IP-range blocking: Google publishes the IP ranges for its fetchers as JSON files, and you can load those into a firewall or WAF to refuse the traffic at the network edge regardless of the user-agent header. The third is emerging: cryptographic bot verification through proposals like Web Bot Auth, which aim to let sites reliably identify and gate automated agents instead of guessing from headers. None of this belongs in robots.txt, and none of it is free — blocking a fetcher that many readers use is a real trade-off, not a default. Ad-supported publishers who live on pageviews have a defensible case to block; brands that want to be named inside AI answers usually want the opposite, to be fetched even when there is no click. Decide by business model, not by reflex. The crawler-blocking-versus-visibility trade-off is worth reading in full before you turn anything off.
Step back and the crawl-control debate resolves into a single strategic fact: you no longer control the destination your content is read at. Google's core crawler obeys robots.txt but feeds AI Overviews you cannot separately exit. A growing class of fetchers ignores robots.txt entirely. Third-party AI crawlers do their own thing. Across all of them, the common outcome is that your content gets read, summarized, and answered somewhere you do not own, increasingly with no click back to you. Fighting that one crawler at a time — with 403s and IP blocks and constantly-renamed user-agent rules — is a defensive game you cannot win, because the number of surfaces is growing faster than your block list.
The move that actually compounds is to stop depending on a crawler to carry your content to an audience and start putting the content directly onto every surface yourself. If a machine is going to read and repackage a single blog post into an answer, the answer is not to hide the post — it is to also exist as a native video on the platforms people scroll, a carousel in the feeds, a newsletter in inboxes, and a presence the answer engines can cite. Discovery has become a distribution problem, not a ranking problem, and AI visibility now means being present everywhere answer engines look rather than winning a single crawl. The publishers who are fine with robots.txt no longer protecting them are the ones who were never relying on a single crawled page for reach in the first place.
Everything above points at one conclusion — the durable answer to "an AI fetcher read my page and sent no traffic" is not a better block rule, it is being present on the surfaces people actually are, in the native format each one rewards. That is the specific problem Kompozy is built for. It is a content generation and multi-platform publishing engine — not a crawler-management tool and not a repurposing add-on — that takes one source idea and produces format-native pieces for every surface, then publishes them directly. Instead of one blog post sitting on your domain hoping a crawler carries it somewhere useful, the same substance ships as short-form and avatar video, carousels, image posts, quote graphics, a blog article, and an email newsletter, fanned by Autopilot across eight social platforms plus blog and email. You are placing the content onto the destinations directly, not waiting to be indexed and summarized on someone else's.
This reframes the robots.txt question entirely. The crawl-control layer is a defensive fight over who is allowed to take your one canonical page; Kompozy plays the offensive game of making that page just one of a dozen native presences across the surfaces where discovery actually happens now. The full set of output formats covers video, image, and text as net-new generation, so a single topic becomes a real post on each platform rather than a link back to a page an AI answer already replaced. When your reach does not hinge on a crawler honoring a robots.txt line, it does not matter as much which fetchers honor it — you are already where the audience is. The honest scope: Kompozy does not stop Google's fetchers, change AI Overviews, or give you a robots.txt lever that does not exist. What it changes is your dependence on any of that. It moves your distribution off the single crawled page and onto the platforms directly, which is the one response to zero-referral AI reading that you fully control. For the measurement side of the same shift, see running AI search visibility as a growth channel; for the economics of why mass-produced pages get diminishing returns from crawling in the first place, scaled AI content and crawl economics.
Google is not broadly ignoring robots.txt for AI, and believing it is leads to the wrong fix. Googlebot, which powers both Search and AI Overviews, still respects robots.txt, and Google-Extended still cleanly opts you out of AI training and grounding with no search cost. The genuine exception is Google's user-triggered fetcher class — Google-GeminiNotebook and its siblings — which ignores robots.txt by documented design, because Google treats a user-requested fetch as a person visiting through a tool, and because robots.txt was only ever a voluntary request, not an enforceable rule. The real SEO tension is not those fetchers; it is that you cannot exit AI Overviews without disallowing the crawler that also keeps you in search. If you must block a fetcher, do it at the server or network layer, by business model, not by reflex. But the strategic conclusion the whole debate keeps circling is simpler: when machines you cannot gate read and answer for your content, the winning move is not to guard a single crawled page — it is to publish your content natively onto every surface yourself, so your reach never depended on that crawler in the first place.
Partly. Googlebot — the crawler behind Search and AI Overviews — still respects robots.txt, and Google-Extended, the robots.txt token for AI training and grounding, respects it too. Neither ignores your rules. What genuinely ignores robots.txt is a separate, growing class called user-triggered fetchers: bots that fetch a page because a person asked a Google product to, such as Google-GeminiNotebook and Google Read Aloud. Google's own crawler documentation states that because the fetch was requested by a user, these fetchers generally ignore robots.txt. So the honest answer is that Google's core SEO crawlers obey robots.txt and one specific category of on-demand fetchers does not.
No, and this is the central SEO tension. AI Overviews are generated from the same index that Googlebot builds for Search — there is no separate AI-Overviews crawler or opt-out token. The only way to keep your pages out of AI Overviews with robots.txt is to disallow Googlebot, which also removes you from ordinary search results. You cannot separate the two. Google-Extended does not help here either: it controls training and grounding in Gemini apps and Vertex AI, not eligibility for AI Overviews. If you want out of AI Overviews specifically, robots.txt is not the tool, and the trade-off is losing organic search entirely.
Google-Extended is a robots.txt token Google introduced in September 2023 — not a separate crawler. Disallowing it tells Google not to use your content to train its generative models or to ground answers in Gemini apps and Vertex AI. Crucially, Google states it has no effect on Google Search: your rankings and indexing are unchanged, because the crawling is still done by Googlebot. It is the one clean lever Google gives publishers to separate AI-training use from search visibility, and it works precisely because it is respected. It does not, however, control AI Overviews or the user-triggered fetchers.
Because Google classifies them as acting on behalf of a specific user rather than as autonomous crawlers. robots.txt was designed to govern automated crawling; Google's position is that when a person points a product like Gemini Notebook at a URL, the fetch is that user visiting the page through a tool, not a bot indexing the web, so robots.txt does not apply. Underneath that is a technical reality: robots.txt has never been an enforceable directive — it is a voluntary request that well-behaved crawlers honor. A fetcher that chooses not to obey it is not breaking a rule, because there was no rule with teeth in the first place. Controlling these fetchers requires server-level or network-level blocking.
At the server or network layer, not in robots.txt. You can match the fetcher's user-agent string in your server config or .htaccess and return a 403, though user agents can be omitted or spoofed. A stronger method is IP-range blocking: Google publishes the IP ranges for its fetchers as JSON files you can load into a firewall or WAF rule. An emerging option is cryptographic bot verification via the Web Bot Auth proposal. Which you choose depends on your model — ad-supported publishers who need the pageviews have a real case to block, while brands that want to be cited in AI answers may prefer to be fetched even without a click.
Mostly no, with one real exception. Googlebot — which crawls for Search and AI Overviews — obeys robots.txt, and Google-Extended is a robots.txt token that opts content out of Gemini training and grounding with no search impact. But Google's user-triggered fetchers, like Google-GeminiNotebook, ignore robots.txt by design, because Google treats a user-requested fetch as a person visiting through a tool. robots.txt is a voluntary request, not an enforceable rule, so blocking those fetchers requires server-level or IP-range controls instead.
Get started → · ← All guides · Compare Kompozy vs other tools