Opt out of AI training without losing search: block the training crawlers, keep the retrieval crawlers, and verify you are still findable in AI answers.
Last verified · 2026-09-16 · by Moe Ameen
The mistake most people make when they opt out of AI training is not opting out — it is opting out with a sledgehammer. A single "block all AI bots" rule feels thorough and quietly deletes you from ChatGPT, Perplexity, Claude search, and Google's AI answers, because it catches the crawlers that surface you along with the ones that train on you. This tutorial does the same job with a scalpel: it blocks the training crawlers by name, keeps the crawlers that keep you findable, and — the part almost every guide skips — verifies that a search crawler can actually still read your site afterward.
Hold one fact in view before you start: the crawlers reading your site do different jobs. Training crawlers absorb your content into a model with no link back. Search-index and answer-retrieval crawlers exist to surface and cite you. They are separate user-agents you control independently, which is the whole reason "opt out of training" and "stay discoverable" are compatible rather than opposed. The strategy behind these steps — the four crawler classes and what discoverability means across surfaces — is in the companion guide, [AI training opt-outs while staying search-discoverable](/guides/ai-training-opt-out-while-staying-search-discoverable). For the broader opt-out checklist across chat tools and platforms, see [how to opt out of AI training on your content](/how-to/opt-out-of-ai-training-on-your-content).
Opt-out mechanisms carry different weight in different places. In the EU, the copyright text-and-data-mining framework treats a properly expressed, machine-readable opt-out as legally meaningful — content may be mined by default unless rights are reserved, and general-purpose AI providers are expected to respect a valid reservation. In the US there is no equivalent statute, so robots.txt directives and platform toggles are largely voluntary or contractual rather than a legal right, and disputes run through the courts case by case. This area moves fast; confirm current behavior in each provider's own crawler documentation before relying on any single control.
Finish the config above and you have confirmed one thing: you are eligible to be found. That is not the same as being cited. A retrieval crawler can only quote a page worth quoting, and when it chooses between eligible sources it favors the one with the broadest, most current coverage of the question — so the payoff of a careful opt-out only lands if there is genuine substance on the surfaces those crawlers watch. The last two steps of this tutorial pointed at the gap: the citations you can earn live mostly on third-party surfaces your robots.txt cannot govern, and being present there is a production job, not a settings job.
Kompozy is a content generation and multi-platform publishing engine built for exactly that half. From one idea it generates 18 output formats — Persona Shorts and other avatar and clipped video, carousels, images, blogs, newsletters, and text — then schedules and fans them across the eight primary social platforms plus blog and email in a single pass. The tie to this task is direct: those are the surfaces answer engines crawl independently of any file on your domain, so publishing genuinely on-brand content on them is how you turn the eligibility you just protected into coverage across the query space where citations are decided. A Persona Brief holds your voice and banned-phrase rules across every asset, so a model reading you on a dozen surfaces assembles one consistent picture, and Autopilot keeps the queue full behind a per-post review gate so the surfaces stay current rather than crawlable-but-stale.
To be clear about the boundary: Kompozy does not write your robots.txt or configure your firewall — that crawler-sorting and verification work is yours, and this tutorial is how to do it. What Kompozy removes is the production constraint that otherwise leaves your protected eligibility with nothing to attach to. Creator ($49/mo for 2,500 credits) fits a solo creator keeping a consistent cross-platform cadence; Pro ($299/mo for 18,000 credits) covers a brand or small team publishing across every channel weekly; Enterprise is custom for agencies running many brands.
No, as long as you only block training crawlers. GPTBot, ClaudeBot, and CCBot have nothing to do with Google. Google-Extended is a training token separate from Googlebot, so blocking it opts you out of training Gemini without touching your search ranking or removing you from results. Rankings only move if you block Googlebot itself, which is the crawler that builds the search index — so never disallow it.
Give each training crawler its own User-agent block with Disallow: / — GPTBot, Google-Extended, ClaudeBot, CCBot, and optionally Applebot-Extended and Bytespider. Then add explicit blocks for the crawlers you keep — Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot — with Allow: / so your permission is unambiguous. Avoid any wildcard "User-agent: *" rule aimed at AI, because it collapses the distinction and sweeps the retrieval crawlers in with the training ones.
Do not trust that the page loads in your browser — you pass challenges a crawler cannot. Check your server and WAF logs for the retrieval-crawler user-agents receiving non-200 responses, and fetch your key pages with each allowed user-agent string to confirm they return real content rather than a challenge page or a JavaScript-only shell. If a search crawler gets anything other than your content, your firewall is overriding your robots.txt and you are invisible to that answer engine.
No. OpenAI runs separate crawlers for separate jobs. GPTBot gathers content that may train foundation models; OAI-SearchBot surfaces sites inside ChatGPT's search and, per OpenAI's documentation, is not used to train those models. Blocking GPTBot while allowing OAI-SearchBot opts you out of training while staying eligible for ChatGPT citations. Anthropic draws the same line between ClaudeBot (training) and Claude-SearchBot (search).
Yes, and more carefully. Since July 1, 2025 Cloudflare has offered blocking AI crawlers by default and exposes granular per-crawler controls. That is powerful, but a default or broad block can catch the retrieval crawlers you want to keep, so use the per-crawler settings rather than an all-or-nothing switch, and verify that Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot are not being challenged. The firewall layer is exactly where a well-meant block quietly removes you from AI answers.