Skip to content
Deepak Gupta markGEO Compass

Glossary · last updated 2026-10-14

AI crawler

Also known as: AI bot, LLM crawler, GPTBot, ClaudeBot, PerplexityBot

A web crawler operated by an AI engine or LLM provider, used to build training corpora and retrieval indexes. The bots whose access controls determine whether your content is reachable to AI engines at all.

AI crawlers fetch web content for two distinct purposes: building training corpora for LLMs (offline, batch) and powering real-time grounding for answer engines (live, per-query). Some user agents do both; some do one or the other. The major user agents publishers track, as documented by their operators:

  • GPTBot (OpenAI): training corpus.
  • OAI-SearchBot (OpenAI): real-time grounding for ChatGPT Search.
  • ChatGPT-User (OpenAI): user-initiated fetches via ChatGPT plugins / browse.
  • ClaudeBot (Anthropic): training only. Blocking it does not remove you from Claude search.
  • Claude-SearchBot (Anthropic): builds the search index Claude uses to answer with web results.
  • Claude-User (Anthropic): fetches pages when a user asks Claude to.
  • PerplexityBot (Perplexity): surfaces sites in Perplexity search results; not used for training.
  • Perplexity-User (Perplexity): user-initiated fetches, which Perplexity says generally ignore robots.txt.
  • Google-Extended (Google): a control separate from Googlebot. Blocking it affects Gemini Apps (training and grounding) but does not affect Google Search, AI Overviews, or AI Mode.
  • Applebot (Apple): crawls for Siri and Spotlight answers that link to sources; see Apple Siri and Spotlight.
  • Applebot-Extended (Apple): a training opt-out signal only; it does not crawl.
  • Bingbot (Microsoft): both classical and AI search via Bing Copilot.
  • CCBot (Common Crawl): third-party crawl used by many LLM training corpora.

Robots.txt controls per-bot access, and what blocking costs depends on the bot's role. Blocking a training-only bot (ClaudeBot, Applebot-Extended) does not remove you from that vendor's search answers. Blocking a search or grounding bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot) can remove you from that engine's answers, a meaningful AI-visibility cost. Blocking Google-Extended affects Gemini Apps but leaves Google Search, AI Overviews, and AI Mode untouched. Most AEO/GEO programmes in 2026 allow all major crawlers and use Cloudflare or similar rate-limiting at the edge to manage abuse without losing visibility.

The economics of opening vs blocking AI crawlers are unsettled and contested. The pragmatic stance: open to the grounding bots (real-time retrieval is the durable AI-visibility surface), opt out of training where the provider honours the signal, and watch the publisher-LLM economic settlement evolve.

Related