Web Bot Auth

How it works

Web Bot Auth is a set of IETF drafts specifying how an automated HTTP client proves which operator it belongs to, using HTTP Message Signatures (RFC 9421). The operator generates a signing keypair, publishes the public key at a well-known HTTPS location, and the agent signs a set of request components with the private key, attaching Signature and Signature-Input headers. An origin or CDN fetches the key, verifies the signature, and learns the operator identity regardless of what the user agent string claims. Three drafts carry it: an architecture document, the HTTP Message Signatures protocol specification, and a registry draft defining the signature agent card and key directory.

When it matters

It matters as soon as your traffic policy depends on knowing which crawler is which, because user agent strings and IP ranges are both forgeable and reverse DNS verification does not scale to agents running on arbitrary infrastructure. No draft had been adopted as a single working group document as of 2026, and it shipped anyway: Cloudflare, Anthropic, and OpenAI moved to production together, which made it the de facto standard ahead of the RFC. See Verify an AI Agent.

Common misconceptions

• "A verified signature means the request is authorized." It authenticates an operator. Authorization is a separate decision you still have to make.
• "It replaces robots.txt." It replaces the guesswork about who is asking. What you permit them to do is still policy.
• "Draft status means wait." Major agent operators and at least one large CDN are in production now. The migration window closes through late 2026.