Most conversations about AI visibility are about real-time answer engines — showing up when someone asks ChatGPT or Perplexity a question today. This guide is about the other half, the part almost no one optimizes for: the training corpus that decides what an AI model knows about you before anyone asks. Common Crawl is the quiet backbone of that layer. It is a free, open, monthly snapshot of the web — billions of pages, more than ten petabytes since 2008 — and it has been the single most-used source of large-language-model training data since the field began, from the C4 dataset behind Google's T5 to the filtered Common Crawl that made up the largest slice of GPT-3's training mix, on through the open models trained today. Its crawler, CCBot, only takes pages it is explicitly allowed to fetch, which means one line in a robots.txt file can silently remove your entire site from the dataset most AI systems learn from — and Common Crawl itself warns that many SEOs block CCBot without realizing they are hiding from AI. This guide explains what Common Crawl actually is, the difference between training visibility and retrieval visibility, exactly how CCBot decides what to include, the blocking mistake that quietly deletes you from the corpus, the three levels at which content reaches AI models, what CCBot can and cannot see, a concrete inclusion checklist, and the honest limit: being in Common Crawl is necessary for baked-in AI knowledge but never sufficient for a citation.
AI visibility has two halves, and almost everyone optimizes only one. The visible half is retrieval: showing up when someone asks ChatGPT, Perplexity, or Google's AI Overviews a question right now. The invisible half is training: what a model already knows about you, baked in before anyone asks a thing. Common Crawl is the backbone of that second half. It is a free, open, monthly snapshot of the web — billions of pages each crawl, more than ten petabytes gathered since 2008 — and it has been the most-used ingredient in large-language-model training data since the field started. If your content is in it, models can learn you exist. If it is not, they largely cannot learn it from there.
The uncomfortable part is how easily you can be left out. Common Crawl's crawler, CCBot, only takes pages it is explicitly allowed to fetch, and it obeys robots.txt. So a single blocking line — often added years ago to save crawl budget, or inherited from a template, or applied as a blanket "keep AI off my site" reflex — can quietly delete your entire domain from the dataset most AI systems train on. Common Crawl itself warns that many SEOs block CCBot without realizing they are hiding from AI. This guide covers what Common Crawl actually is, why training visibility is different from retrieval visibility, exactly how CCBot decides what to include, the blocking mistake, what the crawler can and cannot see, a concrete inclusion checklist, and the honest limit on what any of it buys you.
Common Crawl is a non-profit foundation, started in 2007 by Gil Elbaz, that has been publishing free, open crawls of the web since 2011, with archived page captures reaching back to 2008. Roughly once a month it publishes a fresh crawl — a snapshot of billions of pages, with recent crawls each exceeding two billion URLs — as raw page captures, extracted text, and metadata, hosted publicly so anyone can download and process it. The archive now exceeds ten petabytes and has been cited in thousands of research papers. The point of the project is to democratize access to web-scale data: without it, only companies large enough to run their own planet-scale crawl could build on the web's full text. With it, a researcher or a startup can train on the same raw material as a frontier lab.
That democratization is exactly why it became the foundation of AI training. Google's Colossal Clean Crawled Corpus (C4), built by filtering a single Common Crawl snapshot, was used to train the T5 model family and set the template for the field. OpenAI's GPT-3 paper reported that filtered Common Crawl was the largest single component of its training mix. Meta's LLaMA models were trained on filtered Common Crawl, and a long list of open models continue to lean on it because it is free, enormous, and legally straightforward to use. When people ask "how does an AI model know about the web," a very large part of the honest answer is: because Common Crawl crawled it and someone trained on the result.
Here is the split that determines how to think about all of this. A model's knowledge of your brand comes from two different places. The first is its training data — the baked-in knowledge it absorbed before deployment, which is where Common Crawl lives. The second is retrieval — the live pages an answer engine fetches at query time to ground its answer, which is where real-time crawlers live. These are governed by different bots and different rules, and Common Crawl only touches the first one.
So being in Common Crawl makes a model more likely to know your brand, describe what you do, and mention you from memory when a topic comes up — the "does the model have a mental model of you" layer. It does not, by itself, get you cited in a live answer, because engines like Perplexity, ChatGPT search, and Google AI Overviews retrieve current pages with their own crawlers — GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended — not with CCBot. The mechanics of that live-retrieval side are their own discipline, covered in how Perplexity selects sources and the content formats that actually get cited. Common Crawl is the layer underneath: it shapes what the model brings to the table before it ever searches. You want to win both, but you cannot win them with the same single lever, and the training layer is the one nobody optimizes.
CCBot is a well-behaved, conventional crawler, and understanding its rules is most of the battle. It identifies itself with the user-agent string "CCBot/2.0 (https://commoncrawl.org/faq/)". It checks robots.txt first and honors the Robots Exclusion Protocol, including the crawl-delay directive, which it uses to throttle itself politely rather than as a reason to skip you. It fetches with ordinary HTTP GET requests, follows a few consecutive redirects, and runs from dedicated IP ranges with reverse DNS — legitimate requests resolve under crawl.commoncrawl.org, and the full IP ranges are published so you can verify a request is genuinely CCBot and not an impersonator using its name.
Two behaviors have outsized consequences for inclusion. First, CCBot discovers pages by following links — it works from a frontier of URLs and expands outward through the web's link graph. Pages that nothing links to, or that live behind a form, a search box, or a login, are unlikely to be found and therefore unlikely to be crawled. Second, CCBot does not execute JavaScript. It reads the HTML your server returns; it does not run a browser and wait for a client-side app to render. If your words only appear after JavaScript hydration, CCBot may capture an empty shell and your actual content never enters the corpus. These two facts — link-discoverable, server-rendered — are the practical definition of "crawlable" for training purposes.
The most common way to be absent from Common Crawl is to have told it, on purpose or by accident, to go away. There are three flavors of this. The deliberate block is an explicit rule — "User-agent: CCBot" followed by "Disallow: /" — usually added during a wave of "stop AI from scraping my site" advice. The accidental block is a blanket disallow that was meant for something else but catches every non-Google bot, CCBot included. And the inherited block comes from a CMS template, a security plugin, or a Cloudflare-style bot rule that quietly filters "AI crawlers" as a category. In all three, the outcome is identical: your domain does not enter the crawl, and it is invisible in your analytics because Common Crawl generates no referral traffic — you never learn you opted out of the training web.
This is a genuinely strategic decision, not a reflex, and it is worth making deliberately. Some publishers block AI crawlers on principle or to protect a licensing position, which is a legitimate stance — the economics of that choice, and the emerging option to charge rather than block, are worked through in charging AI crawlers for content access and Cloudflare's AI traffic controls. But blocking to protect content is different from blocking by accident and then wondering why AI models never mention you. And note the asymmetry the broader crawler debate keeps surfacing: some AI companies have been reported to fetch pages even where robots.txt asks them not to, so a block is a weaker guarantee of exclusion than it looks, while it remains a very reliable way to exclude yourself from the compliant, open corpus that Common Crawl represents. The full training-versus-search framing of when blocking helps and when it only hurts is in bot detection vs SEO and what happens when crawlers ignore robots.txt.
Common Crawl frames AI visibility as operating at three levels, and mapping your content to them clarifies what to fix. The first is foundation-model training: the massive pretraining pass where Common Crawl's corpus is a core ingredient. Being here is what gives a model durable, baked-in knowledge of your brand — but it is also the slowest and least current layer, because a model trained last year knows the web as Common Crawl saw it before then. The second is fine-tuning: narrower training on curated datasets, where clean, well-structured content and schema markup make your material more likely to be selected and correctly parsed. The third is real-time retrieval — RAG — where an answer engine fetches live pages at query time, which rewards freshness and accessibility over corpus membership.
The strategic reading is that these levels reward the same fundamentals — accessible, well-structured, substantive content — but on different clocks. Foundation training is the long game: get into the corpus now and compound over successive model generations. Retrieval is the fast game: rank and stay fetchable so you can be cited today. Optimizing only for retrieval leaves the durable training layer to chance, and optimizing only for training ignores the surface where citations actually happen. The through-line for treating all of this as a measurable channel rather than a hope is in AI search visibility as a growth channel and AI visibility measurement.
Being precise about the crawler's blind spots tells you where your content needs to live. CCBot can see public, server-rendered HTML that something links to: your blog posts, articles, product and about pages, documentation, anything reachable by following links and returned as text in the initial HTML response. It reads that text, and increasingly the structured data around it, and that is what enters the corpus.
CCBot cannot see, or sees poorly: content that only renders client-side after JavaScript; pages gated behind logins, paywalls, or forms; anything on a platform it does not log into; and — critically for creators — the substance of video and social posts. It does not transcribe a YouTube video, read the words inside a TikTok or Reel, or crawl the interior of a walled social platform. This is the single most important consequence for anyone whose output is mostly social: an idea that exists only as a video or an ephemeral post is effectively invisible to the training web, no matter how many views it gets. To become part of what AI models learn, the idea has to also exist as crawlable text on the open web — which is exactly the gap most creators never close. The deeper argument for why AI visibility now lives beyond classic SEO, on surfaces link-ranking alone does not describe, is in AI visibility beyond SEO.
Start with access. Open your robots.txt and confirm CCBot is not blocked — no "User-agent: CCBot / Disallow: /" rule, and no blanket disallow or bot-filter that catches it. If you use a Cloudflare-style AI-bot toggle, check whether it is filtering Common Crawl and decide that on purpose. If you have deliberately blocked it for licensing reasons, that is a valid choice — just make it knowingly, not by inheritance.
Then make the content crawlable and parseable. Publish the words as server-rendered HTML so they are present in the initial response, not summoned later by JavaScript. Make pages link-discoverable: internal links, a clean sitemap, no important content stranded behind forms or search boxes. Add schema.org structured data — FAQPage markup is the one Common Crawl explicitly highlights, and article markup helps too — so the text is easy to extract and classify. And most of all, give the crawler something worth having: specific, substantive, genuinely-authored content, because the same specificity that earns AI citations is what makes a page valuable in a training set. That discipline is its own topic in why specific, detailed content gets cited more and AI SEO writing. Finally, accept the clock: crawls are monthly and the models trained on them lag by months more, so treat corpus inclusion as a compounding asset you start early, not a switch you flip for this week's launch.
It would be a disservice to sell Common Crawl as a magic entry point, so here is the honest boundary. Being in the corpus is necessary for a model to learn about you from training data, but it is nowhere near sufficient for a citation. The corpus is billions of pages; inclusion is table stakes, not distinction. Whether a model actually surfaces you depends on how often and how clearly your brand and your claims appear across that data, how much other content corroborates them, and — for live answers — on the entirely separate retrieval and ranking layer. Common Crawl gets you into the room. Being specific, consistent, and widely-referenced across the web is what gets you noticed inside it.
There is also a lag and a control limit worth naming. You cannot force a crawl, you cannot see your Common Crawl inclusion in any analytics dashboard, and the payoff arrives on the training cycle's timescale, not the campaign's. This is why AI visibility is a maintained, long-horizon asset rather than a one-time task — the same logic that governs AI citations, brand mentions, and content refresh. The move is to keep publishing accessible, substantive, crawlable content consistently, so that every monthly crawl and every model generation picks up a stronger, clearer signal of who you are.
The honest framing first. Kompozy does not run Common Crawl, cannot add your site to a specific crawl, and cannot make a model cite you — no tool can, because the corpus and the models belong to other organizations. What Kompozy does is close the exact gap this guide keeps pointing at: most creators pour their best ideas into video and social, the surfaces CCBot cannot read, and never give those ideas a crawlable, text-based home on the open web where they could actually enter the training corpus. Kompozy makes producing that home automatic rather than an extra chore you skip.
Because Kompozy is a generation and multi-platform publishing engine, the same idea you are already turning into Persona Shorts, Clipped Shorts, and carousels also becomes a full Blog Article — server-rendered HTML text, the precise format CCBot ingests — published to your own site through the GHL Blog, WordPress, or a custom webhook. So the substance that would otherwise live only inside a video gets a permanent, linkable, crawlable page that a monthly Common Crawl pass can pick up and a model can learn from. One source idea becomes both the social footprint that earns reach today and the text footprint that compounds into training-corpus visibility over time.
The quality of that text is the part that matters for AI, and it is governed, not generic. Every Blog Article and Email Newsletter is written under a Persona Brief plus banned-word filters, so your pages carry a specific, consistent voice and a clear, unambiguous account of who you are and what you do — exactly the signal that makes a page worth including and easy for a model to associate with your brand. A per-post human review pass is where a person adds the concrete number, named example, or first-hand detail that makes content genuinely authored rather than filler. Then scheduling and publishing across the eight social platforms plus blog and email runs on Autopilot. The result is the thing Common Crawl visibility actually requires and most workflows never produce: a steady stream of accessible, substantive, on-brand text on a site you own — so every crawl finds a clearer version of you than the last.
Common Crawl is the quiet foundation of AI visibility. It is a free, open, monthly snapshot of the web, it has been the most-used source of large-language-model training data since C4 and GPT-3, and it is governed by a polite, rule-following crawler that only takes what it is allowed to take. That makes the first move simple and easy to miss: do not accidentally block CCBot, and do give it crawlable, server-rendered, schema-marked text to read — because a single robots.txt line can delete you from the corpus most AI models learn from, invisibly. But the bigger lesson is structural. The ideas most creators make live only on social and video, where the training web cannot see them. To be part of what AI knows, an idea needs a permanent home in crawlable text on a site you own. Get into the corpus, keep feeding it specific and consistent content, and treat it as a compounding asset — being known by AI is built slowly, one monthly crawl at a time.
Common Crawl is a non-profit that has published a free, open snapshot of the web roughly once a month since 2011 — each recent crawl holds more than two billion pages, and the archive tops ten petabytes. It matters because that corpus is the single most-used source of training data for large language models: Google's C4 (used to train T5) was built from it, filtered Common Crawl was the largest component of GPT-3's training mix, and countless open models train on it today. What ends up in Common Crawl is a large part of what AI models learn the web contains — so being in it is foundational to whether a model knows your brand at all.
It can, and Common Crawl says many site owners do it without realizing. CCBot is Common Crawl's crawler; it obeys robots.txt, so a "User-agent: CCBot / Disallow: /" rule — or a blanket block that catches it — removes your site from the corpus most AI models train on. Because Common Crawl feeds pretraining rather than live search, the effect is invisible in your analytics: you simply never enter the dataset, and models never learn you exist from it. If your goal is to be known and cited by AI, check that CCBot is allowed before anything else.
Allow CCBot in robots.txt (or at minimum do not block it), and publish content it can actually read: server-rendered HTML text, since CCBot does not execute JavaScript, so anything that only appears after client-side rendering may be invisible to it. Make pages linkable and discoverable — Common Crawl finds pages by following links, so orphaned pages and content buried behind forms or logins rarely get crawled. Add schema.org structured data (FAQPage and article markup) so the text is easy to parse. Then be patient: crawls are monthly and models trained on them lag, so inclusion is a slow, compounding asset, not an instant switch.
No, and conflating them is the common mistake. Common Crawl mainly feeds a model's baked-in training knowledge — what it "knows" before it searches. Real-time answer engines like Perplexity, ChatGPT search, and Google AI Overviews retrieve live pages using their own crawlers (GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended), not CCBot. So Common Crawl inclusion makes a model more likely to know and mention your brand from memory, while a live citation depends on those separate retrieval crawlers and on ranking well when the query is asked. You want both; they are governed by different bots and different rules.
Largely no. CCBot crawls the open, linkable web as HTML — it does not log into platforms, and it does not transcribe video or ingest the text buried inside a TikTok, Reel, or YouTube upload. A brilliant idea that only ever exists as a social video or an ephemeral post is close to invisible to the training corpus. The durable move is to also publish that idea as crawlable text on a site you own — a blog post, an article — so the substance you are already creating for social has a permanent, CCBot-readable home that can actually enter the dataset AI models learn from.
Common Crawl is a free, monthly snapshot of billions of web pages, and it is the most-used source of training data for large language models — from the C4 dataset and GPT-3 through today's open models. Its crawler, CCBot, only takes pages it is allowed to fetch, so blocking CCBot in robots.txt quietly removes you from the corpus most AI systems learn from. To stay AI-visible, allow CCBot and publish crawlable, schema-marked HTML text on a site you own.
Get started → · ← All guides · Compare Kompozy vs other tools