Most advice on opting out of AI training frames it as a single lever you pull — and the people who pull it hardest, with a blanket "block all AI bots" rule, quietly delete themselves from the exact place a growing share of their audience now finds them. The two goals sound opposed: keep your content out of model training, and stay findable in Google and in AI answers. They are not, and this guide is about why. The unlock is that your content is not read by one kind of machine but by four, doing four different jobs, and only one of them — the training crawler — is the one you actually want to exclude. The other three are how you get found: the classic search-index crawler that still builds Google's blue links, the answer-retrieval crawler that fetches your page so ChatGPT or Perplexity can quote and link you, and the user-triggered fetcher that pulls a page when a human asks an assistant to read it. Sort those four correctly and opting out of training costs you nothing on discovery. Sort them with a sledgehammer and you protect content nobody can find. This guide lays out the four classes, the block-vs-keep split provider by provider, what "search-discoverable" actually means across the three surfaces you care about, the misconfigurations that silently de-list you even after you get robots.txt right, and the hard limit every domain-level control shares: it only governs the one surface you own, while most of the answers that cite you are assembled from surfaces where you have no robots file at all.
The instinct is understandable: you produce content, models scrape it without asking, so you reach for the switch that keeps them out. The problem is that the switch most people reach for is a blanket "block all AI bots" rule, and in 2026 that rule does something its owner rarely intends — it deletes them from the AI answers and search surfaces where a fast-growing share of their audience now starts looking. So the two things a creator actually wants end up in apparent conflict: keep my work out of model training, and stay findable when someone searches my category. Stated that way they sound like a trade-off. They are not, and the reason is mechanical rather than clever.
Your published page is not read by "AI" as a single actor. It is read by several distinct crawlers run by different companies for different purposes, each identifying itself with its own user-agent string, each controllable independently. Only one of those purposes is the one you object to — a model learning on your work with no link and no credit. The others exist to surface you: to build a search index, to fetch your page so an answer engine can quote and link it, to pull a page when a person explicitly asks an assistant to read it. Once you see that the machines reading you are sorted by job, the whole tension dissolves into a sorting problem. This guide is the sort. It sits between two companion pieces you should read alongside it: the full opt-out landscape, which covers the platform toggles and the hard limits, and bot detection vs SEO, which digs into the firewall side of the same problem. This one stays on the single question of staying discoverable while you opt out.
Start with the taxonomy, because getting this right is 80% of the outcome. There are four functional classes of crawler that matter, and they map cleanly onto four decisions.
These collect content that may be folded into a model's weights, with no attribution and typically negligible referral traffic back to you. The named ones are GPTBot (OpenAI), Google-Extended (Google's training token for Gemini and related products), ClaudeBot (Anthropic), CCBot (Common Crawl, which feeds many downstream datasets), plus others like Applebot-Extended and Bytespider. This is the class the whole exercise targets. Disallowing them is the safe, targeted move — you lose almost nothing you were getting, because these crawlers were never sending you visitors in the first place.
Googlebot is the crawler that builds the classic search index behind blue links, and it is also what Google's AI Overviews and AI Mode draw from. It is not a training crawler and it is not optional: block it and you erase yourself from Google Search and from Google's AI answers in one stroke. Bingbot plays the equivalent role for Microsoft's index and Copilot. This class is pure discoverability infrastructure. Leave it fully allowed, always.
This is the class most people accidentally kill. An answer-retrieval crawler fetches your page so a live AI product can quote it and link back to you — the citation under an answer. OpenAI runs OAI-SearchBot for ChatGPT's search, and OpenAI states plainly that a site opted out of OAI-SearchBot will not appear in ChatGPT search answers. Anthropic runs Claude-SearchBot to index pages for Claude's search feature. Perplexity runs PerplexityBot to fetch the pages it cites. These are the crawlers whose entire job is to send you visibility, and they are separate user-agents from the training crawlers above. Keeping them allowed is how you stay in AI answers while opting out of training.
The fourth class fetches a page only when a human explicitly asks an assistant to read it: ChatGPT-User, Claude-User, and Perplexity-User. They are not automatic crawlers and not training crawlers — they run on demand, one page at a time, when your content is exactly what a real person wanted an assistant to look at. That is a high-intent moment, so unless you have a specific reason to block on-demand reads, leaving these allowed keeps you useful in the increasingly common flow where someone pastes a question and a link into an assistant.
Translate the taxonomy into the named decisions, because the value is in the specifics. The pattern repeats across the three big answer providers: each runs a training crawler you can block and a retrieval crawler you should keep, and the two are different user-agents.
GPTBot gathers content that may train OpenAI's models — disallow it to opt out. OAI-SearchBot surfaces sites inside ChatGPT's search and, per OpenAI's own documentation, is not used to train foundation models — keep it allowed to stay eligible for ChatGPT citations. ChatGPT-User is the on-demand fetcher for when a person asks ChatGPT to read a page; keep it unless you specifically want to block that. Blanket-blocking "OpenAI" collapses all three and throws the visibility away with the training.
Anthropic split its crawlers along the same line and says all of them honor robots.txt. ClaudeBot collects training data — disallow it. Claude-SearchBot builds the index behind Claude's search feature — keep it. Claude-User is the on-demand fetcher for user prompts — usually keep it. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User, so a precise rule opts you out of training while leaving Claude's search able to cite you.
Google draws the line with tokens rather than separate bots. Googlebot is the crawler that builds the index for classic Search, AI Overviews, and AI Mode — leave it fully allowed. Google-Extended is a robots.txt token that controls only whether your content trains Gemini and related generative products; blocking it does not affect your ranking. The catch worth stating plainly, because it trips up publishers who think they have solved this: blocking Google-Extended does not remove you from AI Overviews, because those are assembled from the Googlebot index, not from Google-Extended. There is no clean setting that keeps your full search presence while excluding you from Google's AI answer — being in the index means being usable by it. The mechanics of that bind are worked through in is Google ignoring robots.txt for AI.
Perplexity is primarily a retrieval engine, so PerplexityBot and Perplexity-User exist to fetch and cite rather than to train foundation models — keep them if you want Perplexity citations. The honest caveat: in August 2025 Cloudflare published a report alleging Perplexity used undeclared crawlers with rotating user-agents and IPs to fetch pages that had disallowed it, and subsequently de-listed Perplexity from its Verified Bots program. Take from that the general lesson of this whole area — a robots.txt directive is a request, not a wall, and a crawler that chooses to ignore it faces nothing at the file. Enforcement, if you want it, lives at your server or CDN, not in the text file.
"Discoverable" is not one surface, and being precise about the three it spans is what lets you confirm your opt-out did no collateral damage. The first surface is classic search: the blue-link index, governed by Googlebot and Bingbot. If you only ever blocked training crawlers, this surface is untouched — your rankings do not move, because none of the training crawlers you disallowed had anything to do with the search index. The second surface is AI answers with citations: ChatGPT search, Claude search, Perplexity, Copilot, and Google's AI Overviews. This surface is governed by the retrieval crawlers plus, for Google, Googlebot — keep those allowed and you stay eligible to be quoted and linked. The third surface is the on-demand read: someone hands an assistant your URL. That is governed by the user-triggered fetchers, and it is the surface most people forget exists.
Map your robots.txt against those three surfaces and the discipline becomes concrete. A rule that disallows GPTBot, Google-Extended, ClaudeBot, and CCBot, and explicitly allows Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot, opts you out of the training class while leaving all three discovery surfaces intact. That is the entire trick, and it is why "opt out of training" and "stay search-discoverable" were never really in conflict — they touch different crawlers, and the only way to make them conflict is to stop distinguishing. The step-by-step version of building that file, with verification, is in how to opt out of AI training while staying search-discoverable.
Getting the named rules right is necessary but not sufficient, because two failure modes bypass robots.txt entirely. The first is the blanket rule: a single "User-agent: * Disallow" aimed at AI, or a copied "block all AI bots" list that sweeps the retrieval crawlers in with the training ones. This is the most common self-inflicted wound — it reads as thorough and is actually a decision to disappear from AI answers. The fix is the scalpel above: name the training crawlers, name the retrieval crawlers you keep, and never rely on a wildcard to express "AI."
The second is worse because it is invisible: behavioral bot detection in your WAF or CDN. robots.txt is an explicit, per-user-agent instruction, but a firewall scores requests on behavior — request rate, missing browser fingerprints, whether the client runs JavaScript, IP reputation. Answer-retrieval crawlers look non-human because they are: they fetch fast, often skip JavaScript, and arrive from data-center ranges reputation systems distrust. So a WAF rule tuned to stop scrapers will happily serve OAI-SearchBot or PerplexityBot a challenge page instead of your content, and the crawler records your page as unreadable. Your robots.txt says allowed; your firewall says denied; the answer omits a site it could not read. This failure is common enough that 2026 citation audits keep finding sites that wanted to be cited silently blocking the crawlers they were trying to attract. The full treatment of that trap — including how to test what crawlers actually receive — is in bot detection vs SEO, and the granular per-crawler controls that replace the on/off switch are in Cloudflare AI traffic controls.
Even a perfectly sorted domain runs into a ceiling, and naming it changes where you spend effort. robots.txt controls exactly one surface — the domain you own. But the answers that cite you are overwhelmingly assembled from surfaces you do not control with a robots file: YouTube, LinkedIn, Reddit, TikTok, Instagram, X, the forums and directories where your category gets discussed. Answer engines crawl those independently, and video in particular is one of the source types AI answers lean on most. A brand whose substance exists only on its own site has one crawlable point of presence — and if your robots.txt or WAF is even slightly wrong, it can be zero. A brand present across the surfaces answer engines already trust has many points of presence, and no single domain-level decision can erase it from the answer.
This reframes the goal. Opting out of training on your own domain is the defensive half — the locks you set to keep less of your work in the next training run. Being discoverable is the offensive half, and it is mostly not a robots.txt problem at all; it is a presence problem. The correct opt-out preserves your eligibility to be cited. It does not create the content that earns the citation, and it does not put you on the third-party surfaces where most citations are actually won. That is the seam where a production strategy matters more than any settings page — which is where the next section goes.
Here is the distinction that this guide leaves you with, and it is easy to miss after an afternoon of tuning crawler rules: getting the opt-out right buys you permission to be discovered, not the discovery itself. A retrieval crawler can only quote a page that is worth quoting, and it favors the source that is the most complete and the most current answer to the question — being merely crawlable is table stakes. So the payoff of a careful, discoverability-preserving opt-out only materializes if there is genuine, fresh substance on the surfaces those retrieval crawlers watch. That is a production job, and it is the one the settings page cannot do for you.
Kompozy is built for exactly that half. It is a content generation and multi-platform publishing engine: from one idea it produces 18 output formats — Persona Shorts and other avatar and clipped video, carousels, images, blogs, newsletters, text — then schedules and fans them across the eight primary social platforms plus blog and email. The relevance to this guide is specific: those are the third-party surfaces answer engines crawl independently of your robots.txt, so producing genuinely on-brand content on them is how you convert preserved eligibility into actual answer-presence. And because Autopilot keeps the queue full behind a per-post review gate, the surfaces stay current rather than crawlable-but-stale — freshness being one of the levers that decides which eligible source an engine actually cites.
The consistency layer is what makes the breadth pay off. Every asset Kompozy generates runs through a Persona Brief that fixes your voice, positioning, and banned-phrase list, so a model reading you across a dozen surfaces assembles one coherent picture of who you are rather than a scattered, contradictory one — the corroborated footprint AI answers reward, and the discipline detailed in make content visible to AI search. The boundary is worth stating plainly: Kompozy does not write your robots.txt or configure your firewall — that crawler-sorting work is yours, and this guide is how to do it. What Kompozy does is make sure the eligibility you just protected has something to attach to, on the surfaces where citations are actually earned. For the wider case that visibility is a measurable channel worth running as one, see AI search visibility.
Opting out of AI training and staying search-discoverable were never a genuine trade-off — they only look like one if you treat "AI crawlers" as a single thing to block. They are four things doing four jobs. Exclude the training class (GPTBot, Google-Extended, ClaudeBot, CCBot); keep the search-index crawler (Googlebot), the answer-retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and the user-triggered fetchers. Avoid the blanket rule and audit the WAF that can override your robots.txt behind your back. Then accept the ceiling: your domain is one surface, and the citations you can earn live mostly on the third-party surfaces where robots.txt has no reach — so once the opt-out is set, the work that actually keeps you discoverable is producing real, current content everywhere the answer engines look.
Yes, and it is the whole point of doing it correctly. Opting out of training and staying eligible for AI-answer citation are handled by different crawlers with different user-agents. Blocking the training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) tells models not to learn on your content, while leaving the answer-retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the classic index crawler (Googlebot) allowed keeps you fetchable, quotable, and linkable in AI answers and normal search. The two goals only conflict if you block everything with one blunt rule.
Keep the ones whose job is to surface you, not train on you. Googlebot builds the classic search index and also feeds Google's AI Overviews and AI Mode, so it is non-negotiable. OAI-SearchBot powers ChatGPT's search results, Claude-SearchBot indexes for Claude's search feature, and PerplexityBot fetches pages Perplexity cites — allow all three to stay eligible for AI-answer citations. The user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) run only when a person asks an assistant to read a page, so they are usually worth keeping too.
No. GPTBot is OpenAI's training crawler and has nothing to do with Google. Google-Extended is a robots.txt token that controls only whether your content trains Gemini and related generative products — it is separate from Googlebot, so toggling it does not change your search ranking or remove you from results. The one nuance: blocking Google-Extended does not remove you from AI Overviews either, because those are assembled from the regular Googlebot index, not from Google-Extended.
Training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) collect content that may be absorbed into a model's weights. Search-index crawlers (Googlebot) build the index behind blue links and AI Overviews. Answer-retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch your page so an AI product can quote and link it live. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) pull a page only when a human asks an assistant to read it. You want to exclude the first class and keep the other three.
Only on your own domain, and only for crawlers that honor it. robots.txt (RFC 9309) is a voluntary request with no technical enforcement, so it governs well-behaved bots on the one site you control. It does nothing about the third-party surfaces answer engines lean on most — YouTube, LinkedIn, Reddit, and other platforms they crawl independently. So a correct robots.txt keeps you eligible on your domain, but the majority of the citations you can earn come from being genuinely present on surfaces where you have no robots file at all.
Yes — opting out of AI training and staying search-discoverable are compatible, because different crawlers do each job. Block the training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) in robots.txt while allowing the search-index and answer-retrieval crawlers (Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) that keep you in classic results and AI answers. The mistakes that break this are a blanket "block all AI bots" rule and behavioral WAF rules that silently block the crawlers robots.txt allowed — and the hard limit is that robots.txt only governs your own domain, not the third-party surfaces answer engines cite most.
Get started → · ← All guides · Compare Kompozy vs other tools