Blocking bots used to be a pure win: less scraping, less bandwidth theft, less content lifted without credit. In 2026 that calculus broke, because the same infrastructure that keeps malicious scrapers out — WAF rules, CDN bot-detection, robots.txt, JavaScript challenges — also decides whether the crawlers that feed AI answers ever reach your site. Since Cloudflare made blocking AI crawlers the default on July 1, 2025 and the wider industry followed, a decision that reads as "protect my content" increasingly means "disappear from ChatGPT, Perplexity, and Google's AI answers," where a growing share of high-intent discovery now happens. This guide is about the tradeoff nobody set out to make. It explains why anti-bot systems and AI-search visibility are now in tension, the single distinction that dissolves most of the conflict — training crawlers are not the same as search crawlers, and you can block one while keeping the other — the specific ways bot detection blocks the crawlers you actually want by accident, the Google-Extended bind where the tidy opt-out does not do what publishers think, and how to run a crawler policy that protects your content without deleting your presence from the answers people now trust. The recurring lesson: a blanket block is a blunt instrument in a world that now rewards being precisely readable by the right machines.
For most of the web's history, blocking bots was an unambiguous good. Scrapers cost you bandwidth, lifted your content, and gave nothing back, so a firewall rule that kept them out was pure upside. In 2026 that stopped being true, because the machines reading your site are no longer only adversaries — some of them are the crawlers that decide whether your brand appears in the AI answers people now use to find things. The anti-bot systems built to repel the first kind cannot easily tell them apart from the second, so a blocking decision that reads as "protect my content" increasingly means "delete myself from ChatGPT, Perplexity, and Google's AI Overviews."
That tension became sharp in 2026 for two reasons stacked on top of each other. Discovery moved into AI answers, which raised the cost of being absent from them — the shift documented in AI search behavior is replacing keyword search and AI Overviews are reducing organic clicks. And blocking AI crawlers became a default rather than a chore: on July 1, 2025 Cloudflare became the first major infrastructure provider to block AI crawlers by default, and the rest of the industry moved the same way. This guide is about the tradeoff that created, the one distinction that resolves most of it, and the accidental blocking that catches the people who thought they had it configured correctly.
Start with why anyone blocks AI crawlers at all, because the motivation is legitimate. Training crawlers absorb your content into a model's weights with no link, no attribution, and typically negligible referral traffic in return — Cloudflare's own 2025 analysis measured crawl-to-referral ratios in the hundreds- to tens-of-thousands-to-one for AI crawlers (OpenAI around 1,000 pages crawled per visitor referred, Anthropic tens of thousands) where traditional Google search sits in the single digits. From a publisher's seat that looks like uncompensated extraction: you pay to produce the content, a model ingests it, and the model answers the questions your content used to answer, keeping the visit. Blocking that is a rational defense of a business, and it is the argument behind the crawl economics that broke the old content bargain.
The problem is that the tool for that defense is blunt. When you block "AI bots," the same action tends to catch the crawlers whose entire job is to send you visibility — the ones that read your page so an AI answer can cite it and, sometimes, link a user to you. So the block that protects you from uncompensated training also removes you from the surface where a growing slice of buyers now start. You end up defending the value of content by making it invisible on the channel that is coming to matter most, which is the publisher-traffic bind playing out one firewall rule at a time. The way out is not to pick a side; it is to stop treating "AI crawlers" as one thing.
Here is the single fact that dissolves most of this tradeoff, and it is underused because "block the AI bots" sounds like one decision. It is two. A training crawler collects content that may be folded into a model's weights. A search crawler fetches content so it can be quoted in a live answer with a citation. They are run separately, they identify with separate user-agent strings, and you can allow or deny each one independently. Blocking the first costs you almost nothing you were getting; blocking the second is what actually deletes you from AI answers.
OpenAI runs distinct crawlers for distinct purposes. GPTBot gathers content that may be used to train models. OAI-SearchBot is the one that surfaces sites inside ChatGPT's search feature, and OpenAI states plainly that a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers. The implication is clean: you can disallow GPTBot to opt out of training you get nothing back from, while allowing OAI-SearchBot to stay visible when someone asks ChatGPT a question in your category. Blanket-blocking "OpenAI" collapses both and throws away the visibility with the training.
Google draws the same line with different names. Googlebot is the crawler that builds the search index — it powers classic rankings and it is also what Google's AI Overviews and AI Mode draw from. Google-Extended is a separate robots.txt token that controls only whether your content trains Gemini and related generative products; toggling it does not touch your search ranking. So blocking Google-Extended is the safe, targeted move — you opt out of training without losing a single position. Blocking Googlebot is the nuclear option that erases you from classic Search and AI Overviews at once. The two are not interchangeable, and confusing them is where sites hurt themselves.
There is a catch worth stating on its own, because it trips up publishers who believe they have solved this. Google-Extended lets you opt out of AI training, but it does not remove you from AI Overviews or AI Mode. Those features are assembled from Google's regular search index, which is built by Googlebot — so as long as you are in Search, your content can appear inside Google's AI answers, whether or not you blocked Google-Extended. The closest tool is the nosnippet directive, which does exclude your content from AI Overviews — but it also strips your normal search snippets, so there is no clean opt-out; fully removing your content means keeping it out of the index by blocking Googlebot and forfeiting classic search traffic entirely.
That is a genuine bind, not a configuration error: for Google specifically, there is no setting that preserves your full search presence — snippet and all — while excluding you from the AI answer. Publishers effectively have to accept that being fully in Google Search now means being usable by Google's generative surfaces. It is worth understanding clearly so you do not waste effort on a Google-Extended block expecting it to suppress AI Overviews — it will not — and so you weigh the real choice honestly: index and be used by AI answers, or de-index and lose search. For most brands the answer is to stay indexed and compete inside the answer, which is the strategy in SEO in the age of AI search.
Even after you get the robots.txt right — training crawlers disallowed, search crawlers allowed — you can still vanish from AI answers, and this is the failure that catches careful operators. robots.txt is an explicit, per-user-agent instruction that well-behaved crawlers read and obey. WAF and CDN bot-detection work on an entirely different principle: behavior. They score requests on rate, missing browser fingerprints, whether the client executes JavaScript, IP reputation, and dozens of other signals, and they block anything that looks non-human — regardless of what your robots.txt says.
AI-search crawlers look non-human because they are. They fetch quickly, often do not render JavaScript, and arrive from data-center IP ranges that reputation systems distrust. So a behavioral rule tuned to stop scrapers will happily serve OAI-SearchBot or a legitimate search crawler a JavaScript challenge, a CAPTCHA, or an outright block — and the crawler, unable to solve it, records your page as unreadable and moves on. Your robots.txt said "allowed"; your firewall said "denied"; the AI answer omits a site it could not read. Nobody configured that outcome, which is exactly why it is so common. Broad audits of AI-citation data in 2026 kept finding the same thing: a large share of sites that wanted to be cited had technical barriers — robots rules, CDN policies, JavaScript-only rendering — silently blocking the very crawlers they were trying to attract.
An explicit block is at least visible — you chose it, you can find it, you can reverse it. A behavioral block is invisible from the inside. Your analytics do not show an AI crawler that never got through; your robots.txt looks correct; your pages load fine in your own browser, which sails past the same challenge a headless crawler cannot. The only way to catch it is to test what the crawlers actually receive — check server and WAF logs for AI-crawler user-agents getting non-200 responses, and confirm that your key pages return real content, not a challenge interstitial, to a client that does not run JavaScript. Assume nothing from the fact that the page works for you.
Put the pieces together and the workable policy is precise, not blanket. Decide training separately from search. If you want to opt out of uncompensated training, disallow the named training crawlers — GPTBot, Google-Extended, and any others whose use you reject — in robots.txt. In the same file, explicitly allow the search crawlers you want to stay visible to, above all OAI-SearchBot and Googlebot, so your intent is unambiguous to crawlers that honor the file.
Then audit the layer robots.txt cannot reach. Review your WAF and CDN bot rules against the search crawlers you just allowed, and make sure a behavioral rule is not overriding your explicit permission — allowlist the verified search-crawler user-agents and IP ranges where your provider supports it, and test the response those crawlers get rather than trusting the config. Cloudflare and other providers now expose per-crawler controls and, increasingly, paid-access options like pay-per-crawl for owners who want compensation rather than a binary block; use the granular control instead of the on/off switch. Revisit the policy on a schedule, because the crawler landscape shifts — new engines, renamed user-agents like the Gemini Notebook crawler rename, and changing defaults mean last quarter's allowlist quietly goes stale.
Everything above is about controlling one surface — your own domain — and doing it perfectly is hard precisely because a single misconfigured rule, on a layer you may not even own, can erase you. There is a structural hedge against that fragility, and it is not a firewall setting: it is where your content lives. AI answer engines do not only read your website. They read YouTube, they read Instagram and TikTok and LinkedIn and X, they read the places people discuss your category — and they crawl those surfaces independently of whatever your CDN is doing to your own domain. A brand whose substance exists only on its own site has one crawlable point of failure. A brand present across the platforms answer engines already trust has many, and no single blocking decision can delete it from the answer.
That is the specific reason Kompozy fits next to a crawler policy rather than competing with one. Kompozy is a full generation and multi-platform publishing engine: it takes one idea and produces 18 output formats — Persona Shorts and other avatar and clipped video, carousels, images, blogs, newsletters, text — then fans them across nine social platforms plus blog and email on Autopilot. Video, and YouTube in particular, is one of the source types AI answers cite most, and it is a surface no WAF rule on your marketing site can accidentally block. Producing genuinely on-brand content on all of those surfaces is how you stay readable to answer engines even on the days your own crawlability is broken or contested — the diversification argument laid out in full in AI search visibility as a growth channel.
The consistency piece matters as much as the breadth, because a model reading you across a dozen surfaces is assembling one picture of who you are, and scattered, contradictory descriptions dilute it. Every asset Kompozy generates is written through a Persona Brief that fixes your voice, your positioning, and a banned-phrase list, so the same brand is described the same way everywhere the engines look — the corroborated, entity-consistent footprint AI answers reward. To be clear about the boundary: Kompozy does not configure your firewall or manage your robots.txt — that crawler-policy work is yours, and this guide is how to do it. What Kompozy does is make sure your own domain is not the only place your brand exists to be crawled, so a blocking decision you got wrong on one surface does not take your visibility down with it.
Blocking bots is no longer a free action. In 2026 the same anti-bot infrastructure that keeps scrapers out also governs whether the crawlers behind AI answers can read you, and a blanket block trades content protection for disappearance from the fastest-growing discovery surface. The escape from the tradeoff is precision: block training crawlers you get nothing back from, keep the search crawlers that keep you visible, understand that Google-Extended will not suppress AI Overviews, and audit the behavioral bot rules that silently block the crawlers your robots.txt allowed. Then hedge the whole risk by making sure your brand is present, and consistently described, on the third-party surfaces answer engines crawl no matter what your firewall does. Be readable by the right machines on purpose — that is the entire game now.
It does not hurt classic Google rankings if you block only AI-training crawlers, because those are separate from Googlebot, the crawler that builds the search index. What it does hurt is your AI-search visibility — your presence in ChatGPT, Perplexity, Copilot, and Google's AI answers. Many of those experiences retrieve from live crawls or an index built by specific crawlers, so blocking the wrong one removes you from the answer even while your blue-link ranking is untouched. The risk is losing the fastest-growing discovery surface while your traditional metrics look fine.
A training crawler collects content that may be absorbed into a model's weights — GPTBot (OpenAI), Google-Extended (Gemini training), and the training use of ClaudeBot are examples. A search crawler fetches content so it can be cited in a live, user-facing answer — OAI-SearchBot powers ChatGPT's search, and Googlebot feeds both classic Search and Google's AI Overviews. They are independent systems with separate user-agents, which is the whole point: you can block training you get nothing back from while allowing the search crawlers that keep you visible in AI answers.
Two things happened at once. Discovery moved: a fast-growing share of high-intent queries now resolve inside AI answers rather than a list of links, so being absent from those answers is a real traffic loss, not a rounding error. And blocking got easier and more default: on July 1, 2025 Cloudflare became the first major infrastructure provider to block AI crawlers by default, and the industry followed. So the same year the AI answer became worth appearing in, the tooling made it one click to vanish from it — often without the site owner realizing the two are connected.
Yes, and it is the most common silent failure. robots.txt is an explicit, per-user-agent instruction, but WAF and CDN bot-detection work on behavior — request rate, missing browser fingerprints, IP reputation, JavaScript challenges. A legitimate AI-search crawler that fetches quickly, does not run JavaScript, or comes from a flagged range can trip those rules and get served a challenge page or a block instead of your content. The result is a crawlability failure you never configured: your robots.txt says "allowed," but your firewall says "denied," and the AI answer simply omits a site it could not read.
Be specific instead of blanket. In robots.txt, disallow the named training crawlers you want to opt out of — GPTBot, Google-Extended, and any others — while explicitly allowing the search crawlers, notably OAI-SearchBot and Googlebot. Then audit your WAF and CDN so behavioral bot rules do not silently block the search crawlers your robots.txt permits. Test what those crawlers actually receive rather than assuming. And keep your substance present on the third-party platforms answer engines crawl independently, so no single blocking decision on your own domain can erase you.
Blocking AI crawlers protects your content but can quietly erase your AI-search visibility, because the anti-bot systems that stop scrapers also stop the crawlers that feed ChatGPT, Perplexity, and Google's AI answers. The distinction that resolves most of the conflict is that training crawlers (GPTBot, Google-Extended) are separate from search crawlers (OAI-SearchBot, Googlebot): you can block the first while allowing the second. The trap is behavioral bot detection in your WAF or CDN, which blocks the search crawlers you meant to keep — a crawlability failure you never configured.
Get started → · ← All guides · Compare Kompozy vs other tools