robots.txt for AI Crawlers: The Full Bot-by-Bot Setup

Configure robots.txt per bot, not per company. Every major AI lab ships multiple, independently governed crawlers, and blocking one token does nothing to the others: GPTBot collects OpenAI's training data while OAI-SearchBot builds the index ChatGPT search actually retrieves from, and disallowing the first has zero effect on the second. Miss that distinction and you can silently remove yourself from a fast-growing surface while believing you opted out of something else entirely.

Know the bots before you write a rule

Each vendor runs the same three-way split: a training crawler, a search-indexing crawler, and a live user-fetch crawler.

OpenAI also runs OAI-AdsBot, added in 2026 to validate ad landing pages; it has no effect on citations. OpenAI's own developer documentation states plainly that a site disallowing OAI-SearchBot will not appear in ChatGPT search answers. It is the one token in this whole table you almost never want to block if visibility is the goal.

The decision, made explicit

Blocking every training bot removes you from the most durable form of visibility there is: a search-off answer, where the model already knows your brand without looking anything up. Training data is what builds that layer. Most companies competing for category leadership should allow the training crawlers and make the training-data tradeoff a decision, not an accident inherited from a security template.

A common configuration for a company that wants maximum AI search visibility while still opting out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bingbot
Allow: /

Each user-agent token needs its own directive. A rule under ClaudeBot does not extend to Claude-SearchBot, they are parsed as unrelated agents by design.

Verify past the file itself

robots.txt is necessary and not sufficient. Cloudflare, Akamai, and Fastly bot-management rules block AI crawlers by default in some configurations, silently overriding whatever the file says. Check your CDN and WAF bot rules directly, then verify against server logs, not against intent: parse logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Bingbot, CCBot, Bytespider, Applebot-Extended, and Meta-ExternalAgent, and track which pages actually get crawled and how often. This is the only direct observability you have into the retrieval layer, and it is the fastest way to catch a WAF rule quietly undoing your robots.txt.

The other technical prerequisite: content that needs JavaScript to render is invisible to most AI crawlers. Fetch your own page with curl and check whether the facts you care about are present in the raw HTML response. If they are not, that is the highest-priority engineering ticket on this list, ahead of any crawler directive.

Bing Webmaster Tools is free, most teams never touch it, and it feeds Copilot directly and Perplexity partially. Submit your sitemap there the same day you fix robots.txt.

The llms.txt verdict, stated plainly

The evidence against llms.txt as a visibility lever is stronger than most GEO advice admits. Ahrefs studied 137,210 domains and found 28% had published an llms.txt file, and of the logged requests against those files, roughly 27.5% came from SEO and GEO audit tooling versus about 19.5% from AI bots themselves, meaning a meaningful share of "traffic" to the file is other tools checking whether it exists, not engines reading it. SE Ranking separately found 10.13% adoption across 300,000 domains with no measurable citation lift, and Trakkr's analysis of nearly 38,000 AI-cited domains found zero citation advantage from having one. Google's Search Central guidance, updated June 2026, states outright that Search ignores the file entirely.

Ship one anyway if it takes half a day: the agentic browsing layer may consume it later and the cost is close to zero. Do not let a GEO audit tool's warning about a missing llms.txt pull an hour away from crawler access, server-side rendering, or documentation. Those three have evidence behind them; llms.txt does not, yet.

What to actually prioritize, in order

1. Confirm every search-indexing bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) can fetch the page: check robots.txt, then CDN bot rules, then server logs.
2. Confirm the page's key facts survive server-side rendering: curl the URL and read the raw HTML.
3. Submit sitemaps to Google Search Console and Bing Webmaster Tools.
4. Decide, deliberately, whether to allow training bots.
5. Set up crawl-log monitoring so you can see the effect of every later change.
6. Ship llms.txt last, as a half-day afterthought, not a priority.

How AI search works covers what each engine does once it can actually reach your pages. Running an AI visibility audit is how you find out whether the fix worked.