Block AI crawlers in 2026: robots.txt for compliant bots, server or CDN rules for the rest, and the training-vs-search split that keeps AI-search citations.
Last verified · 2026-09-04 · by Moe Ameen
Blocking AI crawlers looks like a single toggle and is actually two decisions on top of each other. The first is whether to block at all — and if so, which bots, because a blanket block quietly removes you from AI search alongside the training scrapers you meant to stop. The second is where to enforce it: a robots.txt line is a polite request that well-behaved bots honor and aggressive ones ignore, while a server or CDN rule is an actual barrier that returns a block whether the crawler asks permission or not. This walks both decisions in order so you end with a policy that stops the scraping you care about without severing the crawlers that send you readers.
One scope limit to hold before you touch a file: any of this governs only a domain you control. It does nothing about your Instagram, TikTok, or YouTube content, which lives on platforms with their own separate AI-data settings and no robots.txt of yours. If you also want to close the training tap on the tools and platforms you publish through, that is a different and larger checklist — the companion tutorial [opt out of AI training on your content](/how-to/opt-out-of-ai-training-on-your-content) covers it.
robots.txt and bot rules are technical controls, not by themselves a legal prohibition — their weight depends on jurisdiction. In the EU, the copyright text-and-data-mining framework makes a machine-readable rights reservation legally meaningful, so a properly expressed opt-out (robots.txt plus protocols like TDMRep) is treated as binding on general-purpose AI providers. In the US there is no equivalent statute, so most crawler blocks are enforced by contract, terms of service, or the crawler's own stated policy rather than a clear legal right, and scraping disputes run through the courts case by case. Confirm current behavior in each crawler's published documentation before relying on any single control.
Here is the quiet limit of everything above: a crawler block is a fence around one yard. It governs the pages on a domain you own and nothing else, while most of a creator's audience — and most of their content — lives on platforms where you cannot set a single robots.txt line. So the block/allow decision protects your owned site, and that is exactly the surface worth having something substantial on. [Kompozy](/) is the engine that keeps that surface fed while running the distribution layer where crawler rules simply don't apply.
Kompozy is a content generation and multi-platform publishing engine, not a repurposing add-on. From one source it generates the long-form [blog article](/glossary/output-buckets), the carousel, the quote graphic, the text and image posts, and the [persona or avatar video](/glossary/persona-shorts) — 18 formats — then publishes them natively across the eight primary social platforms plus blog and email in a single pass. Two things follow that matter to a page about blocking bots. First, Kompozy publishes as your real accounts; it is never another crawler you have to identify and block, and it does not use your content to train models — so scaling output with it does not widen the training footprint your block just narrowed. Second, the retrieval crawlers you deliberately kept in step six only pay off if the pages they fetch are fresh and worth citing, and Kompozy is how a solo operator keeps an owned site and every platform stocked with structured, on-brand content instead of a stale homepage behind a locked door.
A single [Persona Brief](/glossary/persona-brief) holds voice and banned-word rules across every output, and [Autopilot](/glossary/autopilot) keeps the queue moving behind a per-post review gate. The division of labor is clean: your crawler policy decides who may read your owned domain; Kompozy decides how much there is to read and puts native versions everywhere the fence doesn't reach. Creator ($49/mo for 2,500 credits) fits a solo creator maintaining one owned site plus cross-platform presence; Pro ($299/mo for 18,000 credits) suits a brand or team publishing everywhere weekly; Enterprise is custom for agencies running many domains and brands.
Use both, for different jobs. robots.txt is the fastest way to tell compliant bots like GPTBot and ClaudeBot to stay out, and the major companies honor it — but it is a voluntary request with no enforcement, so aggressive crawlers ignore it. A server or CDN rule (a 403 by user-agent, or a Cloudflare AI-bot rule) is the actual barrier that blocks requests before they read a page. Start with robots.txt for coverage, add the server layer for the bots that don't respect it.
Block the training crawlers: GPTBot (OpenAI), Google-Extended, ClaudeBot (Anthropic), CCBot (Common Crawl), and Bytespider (ByteDance). Deliberately keep the retrieval crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, and user-triggered fetchers like ChatGPT-User and Claude-User — because they quote and link you in AI search. The split is the whole point: block the bots that absorb your content into models, keep the ones that send readers to it.
Not if you target the right agents. Google-Extended is Google's training crawler and blocking it has no effect on Google Search — Googlebot is separate. The real risk is a blanket "block all AI bots" rule that also catches the answer-retrieval crawlers behind AI search results, which removes you from that increasingly important surface. Block training agents by name and leave Googlebot and the retrieval crawlers allowed.
Increasingly, yes. Cloudflare offers AI Crawl Control with Search, Agent, and Training categories and a one-move block, available to all tiers including Free. From September 15, 2026, new setups block Training and Agent crawlers by default on ad-supported pages while Search stays allowed. If you use Cloudflare, check your current AI-bot settings directly — the default may already be doing more, or less, than you intend.
Not completely. robots.txt is voluntary and a bad actor ignores it; even server and CDN rules can be probed by spoofing user-agents or rotating IPs, which is why verifying against published IP ranges matters. Blocking meaningfully reduces training exposure and load from cooperative and semi-cooperative crawlers, and pay-per-crawl or WAF rules raise the cost for the rest — but treat it as reducing exposure, not building an impenetrable wall.