robots.txt (for AI crawlers)

robots.txt is the decades-old robots exclusion file served at the root of a domain, listing per-user-agent rules for which paths a crawler may fetch. In the AI era it has become the front-line control for AI crawler access: publishers add User-agent blocks for AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended to allow or disallow them.

The decision is more consequential than it looks, because of the training-versus-grounding distinction: blocking a training crawler opts you out of a model's training corpus (low visibility cost), while blocking a grounding crawler removes you from real-time citation in that engine (high visibility cost). Conflating the two is the most common mistake, and the dedicated decision guide walks through which bot is which and what blocking each one costs.

Two caveats matter. First, robots.txt is honor-system: well-behaved crawlers respect it, misbehaving ones ignore it, so it is a request rather than a wall. Real enforcement of abusive crawling happens at the edge (CDN-level bot management and rate limiting). Second, blocking Googlebot or Bingbot takes you out of classical search entirely, not just AI surfaces, so the AI-specific user agents are the ones to target. Pair a thoughtful robots.txt with an llms.txt that maps your best content for the bots you welcome.