A Markdown file at a site's root that hands AI agents a clean, linkable map of its key pages. The 2026 V2 update adds formal Markdown link relations.
Last verified · 2026-08-18 · by Moe Ameen
llms.txt is a proposed web standard: a Markdown file published at the root of a domain (`/llms.txt`) that gives large language models and AI agents a curated, easy-to-parse map of a site's most important content. Instead of forcing a model to crawl and untangle full HTML — nav bars, ads, scripts, cookie banners — the file offers a short, structured index of the pages that actually matter, in clean text a model reads cheaply and reliably.
The format is deliberately minimal. A valid file opens with a single H1 (the project or site name), followed by a blockquote one-line summary, optional free-text context, and then H2 sections whose bullet lists are hyperlinks to key pages, each optionally annotated with a short note. A conventional "Optional" section flags lower-priority links an agent can skip under a tight context budget. The spec also encourages publishing a Markdown version of each linked page — historically at the same URL with `.md` appended — so an agent that follows a link lands on clean text rather than a rendered page. A related convention, `llms-full.txt`, concatenates a site's full documentation into one file for models that prefer everything in a single fetch.
Jeremy Howard of Answer.AI proposed llms.txt in September 2024, and on August 10, 2026 published V2 — the first revision. V2 adds a second, cleaner Markdown URL pattern (replacing the file extension, e.g. `/page.md`, with `/index.md` for directories) and, crucially, two formal link relations so agents can discover those files without guessing: `rel="alternate" type="text/markdown"` points from an HTML page to its Markdown twin, and `rel="describedby"` points to the llms.txt file that covers it. Both can be delivered as HTML `<link>` elements or an HTTP `Link:` response header. It is important to be precise about what llms.txt is not: it is a proposed convention for agent readability, not an official standard and not a search-ranking signal — the analogue for AI reading is closer to a sitemap or robots.txt than to anything that changes where you rank.
llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024, as generative AI tools started reading the web at scale and hitting the same wall: HTML is built for browsers and humans, and models waste context and money parsing it. The pitch was a companion to the sitemap era — sitemaps tell search crawlers what exists; llms.txt tells AI agents what matters and hands it to them in clean Markdown.
Adoption grew unevenly. By 2026 thousands of sites published a file; documentation platform Mintlify auto-generates one for every site it hosts, and Anthropic, OpenAI, and Google's Gemini team publish llms.txt files for their developer docs. Skeptics pushed back hard in parallel — a widely shared "llms.txt is dead / a dud" argument noted that the major AI crawlers had not publicly committed to reading the file, so publishing one could be effort spent on something nothing consumes. Google's position sharpened the doubt: Search has repeatedly said supporting llms.txt is "not on its agenda." The V2 update in August 2026 was Howard's answer to two years of that real-world friction — its formal link relations exist precisely so an agent can find the clean Markdown deterministically, closing the "where does the file even live" gap that made early adoption feel like shouting into a void. Notably, Chrome's Lighthouse added an "Agentic Browsing" audit that checks whether a site ships the file, a small sign the convention is being treated as infrastructure even where search declines to endorse it.
| Platform | Behavior |
|---|---|
| ChatGPT / OpenAI | OpenAI publishes an llms.txt for its own developer docs, signaling the format is taken seriously internally, but has not publicly committed its crawlers to reading third-party llms.txt files as a ranking or retrieval input. Treat it as readability insurance, not a guaranteed pickup. |
| Anthropic (Claude) | Anthropic publishes llms.txt for its docs. Agentic Claude workflows that fetch a site can benefit from a clean Markdown map, and the V2 rel="describedby" link relation makes that map discoverable from any page the agent lands on first. |
| Google (Gemini / Search) | Split behavior. The Gemini team publishes an llms.txt for its docs, but Google Search has repeatedly said supporting llms.txt is not on its agenda — so it is not a Google ranking signal. Chrome Lighthouse does include an Agentic Browsing check for the file. |
| Perplexity and answer engines | Answer engines that synthesize from live pages read cleaner when handed structured Markdown, so a well-built file can lower the odds your content is mis-parsed or skipped — though, as with the others, no public commitment guarantees it is consumed. |
| Documentation platforms (Mintlify, etc.) | Some hosts generate llms.txt automatically for every site, which is why adoption numbers rose fast. If your docs live on such a platform you may already ship one without having authored it. |
llms.txt sits in an awkward spot: obviously sensible, not yet obviously consumed. The idea is right — HTML is a terrible way to feed a model, and a curated Markdown index is a genuinely better one. But the honest verdict in 2026 is that it is cheap insurance, not a growth channel, and anyone selling it as a ranking lever is overreaching. V2's link relations are the most useful thing to happen to it, because "the agent couldn't find the clean version" was the actual failure mode, not "the format was wrong."
What the debate keeps skipping is the part that decides whether any of this matters: the pages the file points at. An AI agent handed a spotless map to forgettable content still has nothing to cite. The durable move is to make the destination worth the trip — original, well-structured pages with a real point of view — and to keep producing them at a pace a curated index can actually showcase. That is the layer a tool like Kompozy handles: generating on-brand blog articles and newsletters (and the whole social spread from the same source) through a [Persona Brief](/glossary/persona-brief) that strips the generic AI voice, so the content behind your links reads like a person worth quoting. Ship the llms.txt file in an afternoon; spend the real effort on what lives at the other end of every link. See [Generative Engine Optimization](/glossary/generative-engine-optimization) for how that fits the wider AI-visibility playbook.
llms.txt is a proposed web standard: a Markdown file at a site's root (/llms.txt) that gives AI agents and language models a curated, easy-to-parse map of the site's most important pages, in clean text instead of full HTML. It opens with an H1 site name, a blockquote summary, and H2 sections of annotated links, and it encourages publishing Markdown versions of key pages so agents read structured text rather than scraping rendered pages.
V2, published by Jeremy Howard on August 10, 2026, adds a second Markdown URL pattern (replacing the file extension, e.g. /page.md, with /index.md for directories) and two formal link relations so agents can discover the files: rel="alternate" type="text/markdown" points to a page's Markdown version, and rel="describedby" points to the llms.txt file. Both work as HTML <link> elements or an HTTP Link: header.
It is a cousin, not the same thing. robots.txt tells crawlers what they may access; a sitemap tells search engines what exists. llms.txt tells AI agents what matters and hands it to them in clean Markdown. All three are root-level files aimed at machines, but llms.txt is specifically about giving language models readable, curated content — and unlike robots.txt it is a proposed convention, not a widely enforced one.
Some ecosystems treat it seriously — Mintlify auto-generates it, and Anthropic, OpenAI, and Google's Gemini team publish llms.txt for their own docs — but no major AI crawler has publicly guaranteed it reads third-party files as a retrieval or ranking input, and Google Search says it is not on its agenda. The realistic framing is low-cost readability insurance rather than a guaranteed traffic channel.
No. It is not a ranking signal for Google or any search engine; Google has explicitly said supporting it is not on its agenda. Its purpose is to make your content cleaner and cheaper for AI agents to read and cite. Any ranking benefit comes indirectly, from the quality of the pages it points at, not from the file itself.