Announced October 30, 2026, the three companies agreed to read a single robots.txt line — tied to a new IETF draft — as a signal to keep a site's content out of AI-model training, while leaving ordinary search crawling untouched.
2026-11-01 · by Moe Ameen
On Friday, October 30, 2026, Google, Microsoft, and OpenAI each said they would honor a single, shared line in a site's robots.txt file that signals the publisher does not want its content used to train AI models. For years, opting out of AI training meant maintaining a different rule for every company's crawler — Google-Extended for Google, GPTBot for OpenAI, and so on. The pitch here is one rule that all three read the same way, so a publisher can express the preference once instead of chasing each vendor's token.
The mechanism is a dedicated robots.txt user-agent token that names AI training specifically (reported as `__usage_ai-training`), used inside an ordinary Allow/Disallow rule group — no new directive, so a site could disallow training site-wide but allow it on, say, a press-kit folder. It rides on an Internet Draft submitted to the IETF's AI Preferences (AIPREF) working group by Anthropic's Kevin Kelley, building on the group's in-progress vocabulary draft. In that vocabulary, "AI training" means using content to change a generative model's learned parameters; non-generative uses such as classifiers and ranking systems are out of scope. The token is a preference signal, not a crawl block — it does not stop crawling, and an existing wildcard `User-Agent: *` rule does not count as a training preference. The draft is an individual IETF submission, not a ratified standard.
The three companies are rolling in on their own timelines. Google's John Mueller said that if a site uses the shared line without separately naming Google-Extended, Google will treat it as a Google-Extended block (which also covers grounding in Gemini apps), with support targeted for early 2027; Google-Extended does not affect Search inclusion. Microsoft said it will adopt the shared line and add a separate page-level grounding preference, with Bing Webmaster Tools opening a temporary grounding control around mid-December 2026 and the new options honored during crawling in early 2027. OpenAI said it aims to honor the signal in the coming months while its current robots.txt policy stays in effect. Google and Microsoft were both explicit on one point: this controls training (and, for Microsoft, grounding), not whether your pages appear in search.
Because this is brand-new and phased, confirm the exact token syntax, dates, and per-company behavior on each company's own crawler documentation and the IETF datatracker before relying on specifics.
This standard quietly rewards one thing: owning the ground your content sits on. A robots.txt opt-out is a lever only a domain owner can pull — you can set it on your own site and newsletter, never on a social platform where the platform's terms decide. So the move isn't to retreat from the open web; it's to make sure the substantive version of your expertise lives somewhere you control the training preference AND stays discoverable, because the opt-out explicitly doesn't touch search — and, set deliberately rather than defaulted, doesn't have to touch AI citation either.
That is exactly the publishing posture [Kompozy](/) is built to produce. Point it at one source — a talk, a demo, a recurring customer question — and from a single [Persona Brief](/glossary/persona-brief) it generates the owned, searchable assets: a full Blog Article and an Email Newsletter published to a domain and a list you control (via GHL Blog, WordPress, a custom webhook, or Mailchimp), where your robots.txt line governs training on your terms. In the same pass it ships the social set — [Persona Shorts](/glossary/persona-shorts) and avatar video, Clipped Shorts, brand-exact [Carousel Posts](/glossary/hyperframes), and text posts — across eight social platforms plus blog and email, on [Autopilot](/glossary/autopilot) behind a per-post review gate. The result: the canonical answer in your niche is a page you own and can both opt out of training and keep in front of AI answers. If you want to tune the training vs. visibility split, see our guide on [opting out of AI training while staying search-discoverable](/how-to/opt-out-of-ai-training-while-staying-search-discoverable) and the [generative-engine-optimization](/glossary/generative-engine-optimization) playbook, plus our reporting on [Cloudflare's per-purpose AI controls](/news/cloudflare-block-ai-training-allow-search).
On October 30, 2026, all three said they would honor a single shared robots.txt line as a signal that a site does not want its content used to train AI models. It rides on an Internet Draft in the IETF's AI Preferences (AIPREF) working group, using a dedicated training token inside ordinary Allow/Disallow rules, so a publisher can express the preference once instead of maintaining a separate token for every company.
No. Google stated its training opt-out does not affect Search inclusion, and the token does not even stop crawling — it is a usage preference, not a block. It is less clean for AI answers specifically: Google maps the shared line to a Google-Extended block by default, which also opts you out of grounding in Gemini Apps, while Microsoft treats training and grounding as two separate settings.
Not yet. It is based on an individual IETF Internet Draft, not a ratified standard, and each company is adopting it on its own timeline — largely across 2027 (Google early 2027, Microsoft phased from mid-December 2026, OpenAI in the coming months). Honoring it is voluntary, and crawlers that already ignore robots.txt will keep ignoring it. Confirm the exact token and dates on each company's crawler docs before relying on them.
Set the training preference at your own domain's robots.txt, since that lever only exists on sites you control — not on social platforms. Then keep publishing substantive, answer-shaped content (blog articles, newsletters) to that domain so you remain the citable source. Kompozy generates and publishes that owned content from one brief, and our guide on opting out of AI training while staying search-discoverable walks through the split.