Every AI Crawler in 2026: The Reference Table (And Which Ones Your WAF Is Silently Blocking)

There are roughly two dozen AI crawlers that matter in 2026, they fall into three classes with very different consequences, and your security team has probably blocked several of them without telling marketing. Blocking a training crawler is a defensible business decision. Blocking a retrieval crawler removes you from live AI answers, and almost nobody makes that choice deliberately.

TL;DR

• AI crawlers split into three classes: training (feeds model weights), retrieval (feeds live search answers), and user-triggered (fetches a page because a human asked for it right now).
• Cloudflare measured AI crawlers at 20.3% of verified bot traffic in May 2026, with AI-search bots adding another 6.5%. Roughly a quarter of your verified bot load is AI.
• GPTBot is the most-blocked AI crawler, appearing in 5.52% of DISALLOW rules in Q1 2026, ahead of CCBot at 5.08% and ClaudeBot at 4.88%.
• On September 15, 2026, Cloudflare sets new defaults across three AI traffic categories. Training and Agent get blocked by default for new domains on ad-displaying pages. Search stays allowed.
• The highest-value action for a security vendor is narrow: allow every retrieval and user-triggered bot, then decide about training separately.

The three crawler classes

Almost every bad decision about AI crawlers comes from treating them as one category. They are not. A single robots.txt line that blocks "AI bots" collapses three separate business questions into one answer, and the answer is usually wrong.

Class	What it does	What blocking it costs you	Who should care
Training	Collects content that may end up in model weights. Runs on its own schedule, ignores your publishing calendar.	Nothing immediate. Long-term, the model has weaker parametric knowledge of your brand and category.	Legal, content licensing, executive.
Retrieval	Builds the index that AI search features query at answer time. This is what puts you in a cited answer.	Direct and immediate. You stop appearing as a citation in that product.	Marketing, demand gen, product.
User-triggered	Fetches a specific URL because a person pasted it or asked the assistant to read it.	A prospect who is actively researching you gets an error instead of your page.	Sales, marketing.

Cloudflare's traffic breakdown puts training at nearly 80% of AI bot crawling volume, with user-action and undeclared purposes under 5%. The volume sits in the class that matters least for visibility, which is exactly why volume-based blocking decisions go wrong. Teams see the crawl load, block the noisy bots, and take out the quiet ones that were feeding citations.

The reference table

Every operator below publishes its own bot documentation. Where a company documents a bot but not an exact user-agent token, the table says so rather than guessing.

Bot	Operator	Class	robots.txt token	Blocking cost
GPTBot	OpenAI	Training	GPTBot	Low, short term
OAI-SearchBot	OpenAI	Retrieval	OAI-SearchBot	High. Removes you from ChatGPT search results
ChatGPT-User	OpenAI	User-triggered	ChatGPT-User	High. Breaks live page reads during research
OAI-AdsBot	OpenAI	Validation	OAI-AdsBot	Only matters if you run ChatGPT ads
ClaudeBot	Anthropic	Training	ClaudeBot	Low, short term
Claude-SearchBot	Anthropic	Retrieval	Claude-SearchBot	High. Claude over-indexes on research queries
Claude-User	Anthropic	User-triggered	Claude-User	High
Googlebot	Google	Retrieval	Googlebot	Critical. Also gates AI Overviews and AI Mode eligibility
Google-Extended	Google	Training and grounding control	Google-Extended	Medium. Opts you out of Gemini training and grounding, not Search
GoogleOther	Google	Generic	GoogleOther	Low. Not tied to a specific product
Google-CloudVertexBot	Google	Customer-requested	Google-CloudVertexBot	Low, unless customers build Vertex agents on your docs
PerplexityBot	Perplexity	Retrieval	PerplexityBot	High. Perplexity is citation-first by design
Perplexity-User	Perplexity	User-triggered	Perplexity-User	High. Documented as generally ignoring robots.txt
Meta-ExternalAgent	Meta	Training	meta-externalagent	Low
Applebot-Extended	Apple	Training control	Applebot-Extended	Low. Does not affect Siri or Spotlight surfacing
Amazonbot	Amazon	Mixed	Amazonbot	Low to medium
Bytespider	ByteDance	Training	Bytespider	Low. Frequently blocked for crawl aggression
CCBot	Common Crawl	Training (indirect)	CCBot	Medium. Feeds many downstream training sets at once

Two entries deserve their own note.

Google-Extended is not a fetching crawler. It is a control token. Nothing arrives at your server identifying itself as Google-Extended. Disallowing it opts your content out of Gemini training and out of grounding, which is the process of supplying pages from the Search index to the model at prompt time. It does not affect Google Search ranking, and Google is explicit that normal indexing rules govern AI Overviews eligibility. If a page is blocked from Google Search, it cannot appear as a supporting link in AI Overviews or AI Mode. That single fact makes Googlebot the highest-stakes entry in the table.

Perplexity-User generally ignores robots.txt. Perplexity documents this openly on the grounds that the fetch is user-initiated rather than automated crawling. Whether you accept that reasoning or not, plan for it: if you want to stop that traffic you need a WAF rule, not a robots.txt line.

What your WAF is doing without asking

This is where the cybersecurity vertical differs from every other industry, and why generic AI-visibility advice fails here.

In most companies, robots.txt is a marketing artifact. In security companies, bot policy is a security control, owned by the security or platform team, configured in Cloudflare or Akamai or a bot-management product, and reviewed against a threat model that treats automated crawling as reconnaissance. Nobody in that review is asking whether OAI-SearchBot needs access. The default posture is deny, and it gets applied at the edge where marketing cannot see it.

Three failure patterns come up repeatedly:

• Managed bot rules that predate AI search. A "block unknown bots" rule written in 2022 catches every bot introduced since. OAI-SearchBot did not exist when the rule was written and gets blocked by a policy nobody has revisited.
• Rate limits tuned for humans. AI crawlers arrive in bursts. A per-IP rate limit calibrated for browser traffic returns 429 responses that a crawler reads as unavailability, and repeated 429s degrade recrawl frequency.
• Platform defaults. Cloudflare has blocked AI crawlers by default for new domains since July 2025. If your marketing site was onboarded after that date, the default was chosen for you.

The tell is a gap between what robots.txt permits and what actually reaches origin. robots.txt is a request. A WAF rule is enforcement. When they disagree, the WAF wins and your robots.txt file becomes a document describing a policy that is not in effect.

The crawl-to-refer ratio, and why it makes this argument hard

Cloudflare published crawl-to-refer ratios by operator, and they explain most of the internal resistance you will meet.

Operator	Crawl-to-refer, all sites	Crawl-to-refer, computer and electronics
Anthropic	50,000:1	8,800:1
OpenAI	887:1	401.7:1
Perplexity	118:1	88:1

Read as a bandwidth ledger, these numbers argue for blocking everything. Anthropic crawls a site thousands of times for every visitor it sends back. An infrastructure team looking only at this table will reach for the block button, and their reasoning is internally consistent.

The reasoning fails because the referral is not the product. In a research-stage AI conversation the outcome that matters is being named as the recommended vendor, and that happens whether or not anyone clicks. A CISO who reads a Claude answer that names your platform and then types your domain directly generates a direct-traffic session, not a referral. The crawl-to-refer ratio counts the cost and misses the value entirely. Anyone arguing bot policy from that table alone is optimizing a metric that was never measuring the thing they care about.

What changes on September 15, 2026

Cloudflare is moving to three AI traffic categories, Search, Agent, and Training, each with its own default. For new domains onboarding to Cloudflare, Training and Agent will be blocked by default on pages that display ads. Search remains allowed by default.

The category split is the important part, and it validates the three-class model above. It also means the conversation with your security team changes shape. Instead of a binary allow-or-block argument, you get a per-category decision, and "allow Search, block Training" becomes a position both sides can sign. That is a much easier internal sell than what most marketing teams have been attempting.

If you run on Cloudflare, audit your current category settings before the date rather than after. Existing domains keep their settings, but any new property your team spins up inherits the new defaults.

A default robots.txt for a security vendor

This is the posture I would ship for most cybersecurity companies. It allows everything that produces citations, allows user-triggered fetches so a prospect reading your page inside an assistant gets your page, and leaves the training question as an explicit, separately reviewable decision.

# Retrieval: allow. These produce citations.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# User-triggered: allow. A human is waiting on this fetch.
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Training: decide deliberately, document the decision.
# Uncomment to opt out.
# User-agent: GPTBot
# Disallow: /
#
# User-agent: ClaudeBot
# Disallow: /
#
# User-agent: Google-Extended
# Disallow: /
#
# User-agent: CCBot
# Disallow: /

Two cautions. First, robots.txt is advisory, so this file describes intent, not enforcement. Verify the edge agrees with it. Second, if you block training crawlers, expect a slower decay rather than an immediate drop: the model's parametric knowledge of your category ages out over successive training runs instead of disappearing at once.

How to verify what is actually reaching you

Four checks, in order of how much they tell you per minute spent.

1. Read origin logs, not the CDN dashboard. Filter on the user-agent tokens in the table above and count requests per bot per week. A retrieval bot with zero hits in seven days is being stopped somewhere upstream.
2. Fetch your own robots.txt from outside your network. Some WAF configurations serve a different file to non-browser clients, which produces a robots.txt that looks correct to you and blocks everything to a crawler.
3. Check response codes, not just request counts. A bot receiving 403 or 429 is being blocked in a way that request-count charts will happily report as traffic.
4. Verify identity against published IP ranges. User-agent strings are trivially spoofed. OpenAI, Anthropic, and Perplexity all publish JSON files of their crawler IP ranges. Perplexity's are at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json; Anthropic publishes at claude.com/crawling/bots.json. Verify before you build an allowlist on a header anyone can forge.

Crawl access is the floor, not the ceiling. Once the bots can reach you, the question becomes whether the pages they reach are structured to be extracted, which is a separate problem covered in crawl budget as an AI visibility problem and in the schema markup guide for security content. If you want to see whether any of this is currently working, the free AEO and GEO visibility checks will tell you inside an hour.

Frequently Asked Questions

Which AI crawlers should I never block?

Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot. These are retrieval crawlers that build the indexes AI search features query at answer time. Blocking any of them removes you from that product's cited answers. Googlebot is the most consequential because it also gates eligibility for Google AI Overviews and AI Mode.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot collects training data. ChatGPT's search feature uses OAI-SearchBot, and live page reads use ChatGPT-User. You can block GPTBot and still appear in ChatGPT search results, provided the other two are allowed. Conflating them is the single most common mistake in AI crawler policy.

What is the difference between Google-Extended and Googlebot?

Googlebot is a real crawler that fetches your pages. Google-Extended is a robots.txt control token with no fetching behavior of its own. Disallowing Google-Extended opts your content out of Gemini training and out of grounding at prompt time. It does not affect Google Search ranking or AI Overviews eligibility, both of which are governed by Googlebot access.

Why is my robots.txt being ignored?

Two likely reasons. Either a WAF or bot-management rule is blocking the crawler before robots.txt is ever consulted, which means your file describes a policy that is not in effect, or the crawler is one that documents itself as user-triggered. Perplexity-User is the clearest example: Perplexity states it generally ignores robots.txt because the fetch is initiated by a person rather than by automated crawling.

How much of my bot traffic is AI crawlers?

Cloudflare measured AI crawlers at 20.3% of verified bot traffic in May 2026, with AI-search bots contributing a further 6.5%, so roughly 26.7% of verified bot activity was AI-related. Your own share depends heavily on industry, and content-dense sites see substantially more.

Should a cybersecurity company block AI training crawlers?

It is a legitimate business decision rather than a technical one, and it belongs to legal and executive rather than to the security team alone. The practical version: block training crawlers if your content is a licensed commercial asset, allow them if your category vocabulary is still being established and you want models to learn it with your terminology in it. Either way, decide it separately from retrieval access, and write the decision down.

What happens on September 15, 2026?

Cloudflare sets new defaults across three AI traffic categories. For new domains, Training and Agent crawlers are blocked by default on pages displaying ads, while Search crawlers remain allowed by default. Existing domains keep their current configuration, so the risk is a newly onboarded property silently inheriting a stricter posture than your main site.