For most of the web's history, a site owner's only lever over AI crawlers was robots.txt — a note taped to the door that well-behaved bots chose to read and everyone else ignored. Cloudflare, which sits in front of roughly a fifth of all websites, spent 2025 and 2026 replacing that note with an actual lock, and then a toll booth. This guide walks through the full stack of AI traffic controls Cloudflare now offers and, more importantly, what each one is genuinely for. It starts with the July 1, 2025 "Content Independence Day" move that made Cloudflare the first major infrastructure provider to block AI crawlers by default and launched Pay Per Crawl, a marketplace that charges crawlers per request using the long-dormant HTTP 402 status code. It covers the September 2025 Content Signals Policy — a machine-readable robots.txt extension that lets you state whether your content may be used for search, real-time AI answers, or model training — and why it is a preference layer, not an enforcement one. It covers the July 1, 2026 expansion to per-category controls that gate Search, Agent, and Training bots separately, the September 15, 2026 default that blocks training and agent crawlers on ad-monetized pages, and the shift from Pay Per Crawl to Pay Per Use, which charges AI companies when your content creates value rather than merely when it is fetched. Then it gets to the decision that actually matters: block, allow, or charge is a monetization and control question about the input side of your business — who gets to read your source content and on what terms — and it is a genuinely powerful new lever. But it does not answer the output-side question of whether your brand still shows up in the AI answers and platform feeds where discovery now happens. This guide separates those two problems cleanly, because conflating them is how site owners end up gating their content and quietly vanishing from the surfaces they most wanted to be on.
For most of the web's history, the only tool a site owner had to manage crawlers was robots.txt — a plain-text file that lists which bots may visit which paths. It has one structural weakness that defines everything about the current moment: it is a request, not a rule. A well-behaved crawler reads it and complies; a crawler that chooses to ignore it faces no technical barrier, because robots.txt was never an enforceable lock, only a sign. That was tolerable when the crawlers reading your content were search engines sending traffic back. It stopped being tolerable when AI crawlers began absorbing content into models and answering the questions your pages used to answer, often returning nothing.
Cloudflare's AI traffic controls are the response, and their significance comes from where Cloudflare sits. As a CDN and security layer in front of a large share of all websites, Cloudflare can act on your crawler choices at the network edge — it can actually refuse a request, or demand payment for it, before the request ever reaches your origin server. That is the difference between a note taped to the door and a lock, and then a toll booth. Over 2025 and 2026 Cloudflare built out a full stack of these controls: default blocking, a paid-access marketplace, a content-use signaling standard, and per-category gating. The rest of this guide covers what each one is, and — the part that gets lost in the headlines — what each is actually good for.
The controls did not arrive all at once, and understanding them as three distinct waves makes the whole system legible. Each wave answered a different question left open by the last.
On July 1, 2025, in an announcement it branded "Content Independence Day," Cloudflare became the first major internet infrastructure provider to block AI crawlers by default. The mechanical change is a flip from opt-out to opt-in: instead of the web's historic default where a crawler may take anything not explicitly disallowed, new sites start in a posture where unverified AI crawlers are blocked unless the owner explicitly allows them. Cloudflare also had AI companies begin declaring their crawlers' purpose — training, inference, or search — so owners could make that allow-or-block decision per use case rather than as one blunt switch.
The same day, Cloudflare launched Pay Per Crawl in private beta — the piece that turns "block or allow" into "block, allow, or charge." Technically, it revives HTTP response code 402, "Payment Required," a status code that has sat mostly unused in the standard for decades. When a verified crawler requests paid content, Cloudflare can return a 402 with a price in a crawler-price header; the crawler can retry agreeing to that price, or declare a crawler-max-price up front and receive the content with a crawler-charged header confirming what it paid. Cloudflare aggregates the events, bills the crawler, and distributes earnings to the publisher, using Web Bot Auth cryptographic signatures so only verified crawlers can transact. It is the first credible attempt to build a per-request market for AI access to content, rather than a binary of extraction or exclusion.
Blocking answers "may this crawler reach my content." It does not answer "and what may it do with the content once it has it," which is a different and increasingly important question. On September 24, 2025, Cloudflare introduced the Content Signals Policy to fill that gap: a machine-readable extension to robots.txt that lets an owner express use preferences, not just access. It defines three signals — search (classic indexing for results pages), ai-input (using your content in real-time AI answers and inference), and ai-train (using it as training data for a model) — each of which can be set to yes, no, or left unset to mean the owner has not expressed a preference. Cloudflare rolled the standard out across millions of the domains it manages.
The essential thing to understand about Content Signals is that it is a preference layer, not an enforcement one. Like everything else in robots.txt, it is a request that compliant crawlers may choose to honor and non-compliant ones may ignore — the same voluntary-compliance limitation that has always applied, examined in depth in the guide on whether Google ignores robots.txt for AI. Content Signals is valuable as a clear, standardized way to state your terms, and as a legal and reputational marker of intent. It is not a lock. The lock is the edge-level blocking and the toll booth is Pay Per Crawl, and Cloudflare is explicit that signals should be paired with those enforced controls when you need teeth rather than a stated preference.
The third wave, announced July 1, 2026, made the controls granular and the marketplace smarter. On the control side, Cloudflare gave all customers per-category management of three bot types: Search (indexing for results), Agent (real-time automated actions taken on a user's behalf), and Training (data collection for model development). This matters because the right answer often differs by category — many publishers want to stay in search results while blocking or charging for training and agent access, and one on/off switch could not express that. Cloudflare also set a new default: starting September 15, 2026, on ad-monetized pages, training and agent crawlers are blocked by default while search crawlers remain allowed, on the reasoning that an ad is a signal the owner intended a human to land there. Those defaults apply to new customers, new sites of existing customers, and all existing free-tier customers; any owner can adjust them before the deadline.
On the marketplace side, Cloudflare announced that Pay Per Crawl is expanding into Pay Per Use — charging AI companies when your content actually creates value rather than every time a page is fetched. The motivation is an efficiency problem in the crawl economy: Cloudflare noted that over half of AI crawl traffic is spent re-fetching unchanged pages, so billing per fetch rewards wasteful crawling. Pay Per Use instead ties payment to outcomes like a publisher's content appearing in an answer or premium content being accessed, with early partners including AI search and answer engines. It is an early-stage shift, but the direction is clear: from "pay to crawl" toward "pay when the content does work." CEO Matthew Prince has framed the whole effort around a blunt fact — the majority of traffic on the internet is now non-human — which is the reality these controls are built to price.
The single most useful lens on this whole stack is the distinction between preference and enforcement, because it sorts the tools into what they can and cannot do. robots.txt and the Content Signals Policy are preference layers: they state your terms in a standard form and rely on the crawler's good behavior. They are worth using — they are the clearest way to declare intent, and increasingly a reference point in licensing and legal conversations — but a crawler that ignores them pays no technical price, which is exactly why less scrupulous scrapers have always been able to route around them, as covered in the broader look at AI crawler blocking and bot detection.
Cloudflare's edge-level blocking and Pay Per Crawl are enforcement layers. Because the request physically passes through Cloudflare's network before reaching your server, Cloudflare can refuse it, challenge it, or hold it for payment regardless of whether the crawler wanted to comply. Verified-bot cryptographic signatures make it hard to simply spoof a friendly user-agent to slip through. That is a categorically stronger position than robots.txt has ever offered, and it is the reason these controls are a genuine change rather than another well-intentioned standard. The practical rule: use Content Signals to state your terms, and use the enforced controls to actually hold the line where it matters.
What these controls really give a site owner is a decision that used to be binary and is now three-way. Historically you could allow a crawler or disallow it. Now you can block it at the edge, allow it freely, or allow it for a price — and, with per-category controls, you can make that choice differently for search, agent, and training access. That flexibility is the whole point, and it maps onto real business situations. An ad-supported publisher absorbing heavy crawl costs for near-zero referral traffic has a strong case to block or charge training crawlers while keeping search open. A niche knowledge site might want to be trained on and cited, and so allow freely. A premium-content business might charge for everything. There is no universal right answer, which is the argument for granular controls over a default.
The economic backdrop makes the decision live rather than academic. Cloudflare's own measurements have shown AI crawlers taking enormous volumes of content relative to the traffic they send back — crawl-to-referral ratios orders of magnitude worse than traditional search — which is the imbalance the crawl economics of scaled AI content examines and the reason so many publishers welcomed a way to charge. At the same time, a meaningful fraction of "AI traffic" is low-value or automated noise rather than genuine demand, a reality worth understanding before you price it, covered in the analysis of fake AI traffic and bot engagement. Blocking, allowing, and charging are all defensible; the mistake is treating the choice as obvious in either direction.
Here is where site owners get into trouble, and it is a conceptual error rather than a technical one. Cloudflare's controls govern the input side of your content business: who is allowed to read the source material sitting on your domain, and on what terms. They are excellent at that. What they do not govern is the output side: whether your brand actually shows up in the AI answers, the platform feeds, and the recommendation surfaces where a growing share of discovery now happens. Those are two different problems, and the controls only touch one of them.
The failure mode is to conflate the two and reach for a blanket block as if it settled both. It does not. Blocking training crawlers while keeping search crawlers allowed generally protects your source content without harming search visibility — but a blunt "block all AI" posture can quietly remove you from the exact AI answer engines you most wanted to be cited in, and being absent from those answers is a real loss now that so many high-intent queries resolve inside an AI response rather than a list of links, a shift documented in how AI Overviews are reducing organic clicks and the collapse of publisher referral traffic in AI discovery. Gating your content and vanishing from discovery is a real, common, self-inflicted outcome. The controls are a scalpel; used as a hammer they cost you the visibility you were trying to monetize.
The deeper point is that your discoverability should not hinge on whether any single crawler is allowed to reach a single canonical page. When it does, every crawler-policy decision becomes a visibility gamble. When it does not — when your presence in AI answers and platform feeds comes from content you have already published directly onto those surfaces — you are free to make the access-and-monetization decision purely on its merits, because your reach no longer depends on it. That is the mindset the smartest publishers are adopting, and it is examined from the measurement angle in running AI search visibility as a growth channel and the strategic angle in SEO in the age of AI search.
Cloudflare's controls and a tool like Kompozy sit on opposite sides of the same coin, and using both is how you get the whole picture. Cloudflare is the input-side gate: it decides who reaches the source content on your domain and whether they pay. Kompozy is the output-side engine: it decides how much finished, native presence that same source becomes across the surfaces where audiences and answer engines actually are. One controls the front door; the other seeds the neighborhood. They are complementary, not competing, and the reason to pair them is precisely the trade-off above — the controls let you gate and monetize your source safely only if your discoverability is coming from somewhere the gate does not affect.
Concretely: Kompozy is a full AI content generation and publishing engine, not a crawler tool and not a repurposing add-on. Point it at the same source material you are now choosing to gate or charge for — a long-form video, a briefing, a blog draft — and it generates net-new content across eighteen formats: persona and avatar video, clipped shorts, carousels, quote graphics, image posts, blog articles, and email newsletters, each genuinely native to its destination rather than one asset copy-pasted around. Then it publishes that output directly, fanning it via Autopilot across eight social platforms plus blog and email, on a schedule, with a human approval step in the loop. Your presence on those surfaces is content you shipped, not content a crawler lifted — which means it exists whether or not you let any given AI crawler through your Cloudflare settings.
That decoupling is the practical payoff. Once your discoverability is driven by content you have deliberately placed onto every relevant surface, the Cloudflare decision becomes a clean business call: block the training crawlers whose extraction you object to, charge the ones you are willing to sell to via Pay Per Crawl, keep search open, and state your terms with the Content Signals Policy — all without gambling your reach, because your reach was never riding on that single crawled page. The honest scope matters: Kompozy does not manage your crawlers, set your Cloudflare defaults, or bill AI companies for access — that is exactly what Cloudflare's controls are for, and you should use them. What Kompozy changes is your dependence on the outcome. It moves your distribution off the canonical page an AI answer might replace and onto the platforms directly, so gating the source no longer means gating your growth.
Cloudflare's AI traffic controls are the most substantial shift in a site owner's power over crawlers since robots.txt itself, because they replace a voluntary request with enforced, network-level choices. Since July 1, 2025, Cloudflare has blocked unverified AI crawlers by default and offered Pay Per Crawl to charge them via HTTP 402; the September 2025 Content Signals Policy lets you state whether content may be used for search, AI answers, or training; and the July 2026 wave added per-category gating of Search, Agent, and Training bots, a September 15, 2026 default blocking training and agent crawlers on ad pages, and the shift toward Pay Per Use. Use all of it — deliberately, per crawler category, matched to your business model. But keep the input-side decision separate from the output-side one. Blocking, allowing, and charging govern who reads your source; they do not, by themselves, keep your brand present in the answers and feeds where discovery now happens. The durable position is to gate and monetize your source with Cloudflare on your own terms, while making sure your reach comes from content you have already published everywhere it needs to be — so the gate protects your content without ever costing you your audience.
They are a set of network-level tools that let a website decide whether AI crawlers can access it, and on what terms. Because Cloudflare sits in front of a large share of the web as a CDN and security layer, it can enforce these choices at the edge rather than merely requesting them. The stack has three main pieces: default blocking of unverified AI crawlers (introduced July 1, 2025), which flips the web from opt-out to opt-in; Pay Per Crawl, now expanding into Pay Per Use, a marketplace that lets publishers charge AI companies for access; and the Content Signals Policy, a robots.txt extension for stating how content may be used. In July 2026 Cloudflare added granular per-category controls that gate Search, Agent, and Training bots separately, so a site can allow search indexing while blocking or charging for training and agent access.
Yes, with important nuance. On July 1, 2025 — a date Cloudflare branded "Content Independence Day" — it became the first major internet infrastructure provider to block unverified AI crawlers by default, meaning new sites start in an opt-in posture where AI crawlers must be explicitly allowed rather than explicitly blocked. In July 2026 Cloudflare went further: starting September 15, 2026, on ad-monetized pages, training and agent crawlers are blocked by default while search crawlers stay allowed. Those 2026 defaults apply to new customers, new sites of existing customers, and all existing free-tier customers, and any owner can change them in their settings. Existing paid customers keep their current configuration unless they opt in.
Pay Per Crawl, announced in private beta on July 1, 2025, revives the long-dormant HTTP 402 "Payment Required" status code. When a verified crawler requests paid content, Cloudflare can return a 402 with a price in a crawler-price header; the crawler can retry agreeing to pay, or send a crawler-max-price header up front and receive the content with a crawler-charged header confirming the amount. Cloudflare aggregates the events, bills the crawler, and pays the publisher, using Web Bot Auth cryptographic signatures so only verified crawlers are charged. In July 2026 Cloudflare announced an evolution to Pay Per Use, which charges AI companies when content actually creates value — for example when it appears in an answer — rather than every time a page is fetched, a response to the fact that over half of AI crawl traffic re-fetches unchanged pages.
The Content Signals Policy, announced September 24, 2025, is an extension to robots.txt that adds a machine-readable way to state not just whether a crawler may access your content but how it may use it afterward. It defines three signals — search (traditional indexing), ai-input (using your content in real-time AI answers), and ai-train (using it to train models) — each set to yes, no, or left unset to mean no expressed preference. The critical difference from Cloudflare's blocking and Pay Per Crawl tools is that Content Signals is a preference layer, not an enforcement one: like the rest of robots.txt, it is a request that compliant crawlers may honor and others may ignore. For teeth, you pair it with the edge-level blocking or paid-access controls, which enforce at the network rather than asking politely.
It depends on your business model, and the two things people conflate should be kept separate. Blocking or charging governs the input side — who is allowed to read the source content on your domain, and whether they pay for it — which is a legitimate and newly-powerful monetization lever, especially for ad-supported publishers absorbing high crawl costs for little referral traffic. But it does not govern the output side: whether your brand appears in the AI answers and platform feeds where a growing share of discovery now happens. Blocking training crawlers rarely hurts search visibility if you keep search crawlers allowed, but a blanket block can remove you from the very AI surfaces you want to be cited in. The durable approach is to make the access-and-monetization decision deliberately per crawler category, and to keep your discoverability from depending on whether any single crawler is allowed to reach your site.
Cloudflare's AI traffic controls let a site block, allow, or charge AI crawlers at the network edge — enforced, not merely requested. Since July 1, 2025 it blocks unverified AI crawlers by default on new domains, flipping the web to opt-in; Pay Per Crawl, now expanding to Pay Per Use, charges crawlers via the HTTP 402 status code; the September 2025 Content Signals Policy adds robots.txt signals for search, AI input, and AI training; and 2026 controls gate Search, Agent, and Training bots separately, blocking training and agent crawlers by default on ad-monetized pages from September 15, 2026.
Get started → · ← All guides · Compare Kompozy vs other tools