OpenAI: Robots.txt May Not Apply to ChatGPT’s Fetch Bot

Share

ChatGPT’s Fetch Bot Is Reaching Disallowed Pages—Here’s What That Means

New data shows ChatGPT’s page-fetching bot is reaching sites that have disallowed it, and OpenAI’s own documentation explains why the file may not stop it.

TollBit’s latest State of the Bots report covers the first half of 2026, tracking how AI crawlers behave across European and North American sites. Here’s what the numbers show and why they matter for your robots.txt strategy.

Which Bots Are Bypassing Robots.txt Most Often

Across the European sites analyzed, roughly 15% of identified AI page-fetchers accessed URLs that site owners had explicitly disallowed.

The behavior is concentrated among a few agents. ChatGPT-User, Bytespider, and Youbot each reached disallowed pages on nearly half of the European sites that listed them. ChatGPT-User accessed the most sites overall.

How Many Sites Actually Disallow These Bots

Most new page-fetching agents face minimal blocking. Only 9% of European websites disallow Claude-User, compared to 26% in North America. Perplexity-User is disallowed by 13% of European sites versus 26% in North America.

Across Europe, most of the newest agents have disallow rates in the single digits. ChatGPT-User is the notable exception, with far more sites choosing to block it.

What OpenAI’s Documentation Says About Robots.txt

OpenAI’s crawler documentation states that ChatGPT-User fetches a page when a ChatGPT user asks a question. Because the action is initiated by a user, the company says robots.txt rules may not apply.

Perplexity takes a similar position, saying Perplexity-User generally ignores the file for the same reason. Anthropic disagrees: it says all three of its bots respect robots.txt, as we reported in February. TollBit counts any request to a disallowed URL as a bypass, regardless of what the bot operator claims.

Why This Matters for Your Site

A disallow rule for ChatGPT-User is a request, and OpenAI’s documentation says that request may not apply.

There’s a more important distinction to note. According to OpenAI, the agent that decides whether your site appears in ChatGPT search results is OAI-SearchBot—not ChatGPT-User. Sites that block both agents to stop AI traffic have given up the visibility half of the deal while keeping a fetching control that comes with a carve-out.

Your server logs or CDN records show what actually arrived. Robots.txt only shows what you asked for.

What Changes With Cloudflare’s New Controls

Cloudflare is moving crawler management to the network layer. For bots it recognizes, compliance is no longer left to the crawler itself. Starting September 15, new domains added to Cloudflare will have Training and Agent crawlers blocked by default on pages that show ads. Search crawlers remain allowed.

Whether the user-initiated loophole survives is an open question. The argument rests on the idea that requesting a page is different from a crawler taking it—yet all major AI assistants now fetch pages this way.

Featured Image: Net Vector/Shutterstock

Read more