OpenAI's crawler documentation now states that because ChatGPT-User fetches a page only when a person asks, robots.txt rules "may not apply" — leaving a Disallow line unable to stop live ChatGPT browsing, unlike GPTBot and OAI-SearchBot.
2026-08-15 · by Moe Ameen
OpenAI's crawler documentation now separates ChatGPT's page-fetching bot from its automatic crawlers and tells site owners that a robots.txt block may not stop it. The company lists four distinct agents: GPTBot, which crawls the open web to train foundation models; OAI-SearchBot, which surfaces sites inside ChatGPT's search feature; OAI-AdsBot, which checks pages submitted as ads; and ChatGPT-User, which fetches a page in real time when a person asks ChatGPT or a Custom GPT to read it. The user-agent string for the fetch bot is ChatGPT-User/1.0, and its IP ranges are published at openai.com/chatgpt-user.json.
The line that drew attention, flagged in an August 14, 2026 Search Engine Journal report, is OpenAI's own wording for ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." The docs add that ChatGPT-User "is not used for crawling the web in an automatic fashion" and steer anyone who wants to manage automatic crawl opt-outs toward OAI-SearchBot instead. The framing treats a user-triggered fetch as closer to a person opening a browser tab than to a crawler indexing the site, which is why the usual crawl-directive file is presented as not necessarily binding on it.
The practical gap is that GPTBot and OAI-SearchBot remain fully controllable through robots.txt, while ChatGPT-User is the one many sites most want to gate — reporting cited by the piece notes ChatGPT's fetch bot is disallowed by more sites than any comparable AI agent. If robots.txt does not govern it, a publisher who wants to keep ChatGPT from reading a page during a live session has to move the control to the server: a firewall, WAF, or user-agent rule that returns a 403 for ChatGPT-User, ideally verified against OpenAI's published IP list rather than the spoofable user-agent alone. This mirrors how Google handles its own user-triggered fetchers, which its documentation likewise says generally ignore robots.txt.
The instinct on a story like this is defensive — how do I block the bot. For most creators the more valuable read is the opposite: if ChatGPT-User is going to fetch and quote your pages whenever someone asks about your topic, the return is on giving it a lot of clear, on-brand, well-published content to find. Earlier reporting already showed [ChatGPT's search index surfaces small sites the same way it does big publishers](/news/chatgpt-search-indexes-small-sites), so visibility is a coverage-and-clarity game, not a domain-authority one. That is a production problem, and production is what [Kompozy](/) does. It generates net-new content across [18 formats](/glossary/output-buckets) — [blog articles](/glossary/output-buckets), newsletters, carousels, quote cards, and persona video — all governed by one [Persona Brief](/glossary/persona-brief) so the voice ChatGPT reads back is consistently yours, then publishes the set across the eight social platforms plus blog and email on [Autopilot](/glossary/autopilot).
The distinction that makes this OpenAI change matter is worth keeping straight: robots.txt still governs GPTBot and OAI-SearchBot, so you keep your normal crawl and training opt-outs; ChatGPT-User is the live-fetch agent you may not be able to gate with a text file, which is exactly why owning your published footprint beats trying to wall it off. Kompozy turns one idea into a week of citable pages and posts — a blog explainer, a LinkedIn take, a carousel, a short — each a surface ChatGPT can fetch, quote, and link back to. Unlike the Google-side case, where the equivalent [Gemini Notebook fetcher also ignores robots.txt](/news/gemini-notebook-crawler-ai-scraping), the practical answer is the same across engines: publish more of the right thing, in your own voice, on a schedule, so that when an AI reads your work it reads the version you shipped.
Not reliably for user-triggered fetches. OpenAI's documentation says that because ChatGPT-User visits a page when a person asks, "robots.txt rules may not apply." A Disallow line still governs GPTBot (training) and OAI-SearchBot (search), but it may not stop ChatGPT-User from fetching a page during a live chat session.
ChatGPT-User (user-agent ChatGPT-User/1.0) is OpenAI's fetch bot that loads a page in real time when a person asks ChatGPT or a Custom GPT to read or browse it. GPTBot is the automatic crawler that collects data to train models. OpenAI treats them differently: GPTBot obeys robots.txt, while ChatGPT-User is framed as user-initiated and may not.
Move the control off robots.txt and onto the server. Add a firewall, WAF, or user-agent rule that returns a 403 for ChatGPT-User, and verify requests against OpenAI's published IP ranges (openai.com/chatgpt-user.json) rather than trusting the spoofable user-agent alone. Note that blocking the fetch can also stop ChatGPT from citing or linking your page.
For most creators, being cited is the better outcome. The same fetch that reads your page is often what lets ChatGPT quote and link it in an answer, and ChatGPT's index surfaces small sites alongside large publishers. Publishing clear, on-brand, well-structured content across platforms — the kind Kompozy generates and schedules — makes you the source the model reads back.