Verify an AI Agent: Web Bot Auth and Signed Agent Traffic

An AI agent hitting your site presents a user agent string, an IP address, and nothing else. All three are trivially forged. Web Bot Auth fixes that by having the agent cryptographically sign each request with a key you can verify against a published directory, which turns "this claims to be ChatGPT" into "this is provably the holder of OpenAI's signing key".

It is the BIMI story repeating one protocol layer down. Self-asserted identity is being replaced by certificate-backed identity, and the same three properties hold: the verification is cheap for the receiver, expensive to counterfeit, and fails silently when misconfigured.

What Web Bot Auth is

Web Bot Auth is a set of IETF drafts, authored primarily out of Cloudflare with a co-author at Google, that specify how automated HTTP clients identify themselves using HTTP Message Signatures (RFC 9421).

The mechanism is simple. An agent operator generates an Ed25519 signing keypair and publishes the public key at a well-known HTTPS location. When the agent makes a request, it signs a set of headers with the private key and attaches Signature and Signature-Input headers. The origin fetches the public key, verifies the signature, and now knows which operator sent the request regardless of what the user agent string says.

Three drafts carry it: an architecture document, an HTTP Message Signatures protocol specification, and a registry draft covering the signature agent card and key directory. None had been adopted as a single working group document as of 2026, and it is shipping anyway. Cloudflare, Anthropic, and OpenAI moved to production in parallel, which is what made it the de facto standard well ahead of the RFC.

The signature proves operator identity, not intent, not authorization, and not that the request is welcome. A verified agent is still an agent. Verification tells you who to apply policy to; it does not decide the policy.

What it replaces, and why that mattered

The previous state of the art was reverse DNS verification: resolve the client IP to a hostname, resolve that hostname forward, confirm it matches. Google published this method for Googlebot and it worked for a decade.

It scales badly for the current situation. Reverse DNS requires every operator to control static IP space and every origin to maintain per-operator verification logic. It also breaks entirely when an agent runs on behalf of an individual user from arbitrary infrastructure, which is the shape of most agentic browsing.

robots.txt never solved this either, for a reason worth stating plainly: it is a request, not a control, and it is addressed to a name the requester chose for itself.

Step 1: Decide what you are protecting

Verification is worthless without a policy to attach to it. Before implementing, write down which of these you actually want:

• Allow verified agents, rate-limit unverified ones. Most common, least disruptive.
• Allow named operators only. Appropriate for paid content or an API surface.
• Block all agents, verified or not. Legitimate, and much easier to enforce correctly once agents are identifiable.
• Allow everything, log the identity. The right first step. You cannot set policy on traffic you have never measured.

Start at the last one. Nearly every team that begins with blocking discovers they blocked a crawler that was driving referral traffic.

Step 2: Take the CDN implementation if you have one

If you sit behind a CDN that implements signed agent verification, turn it on there and stop. The edge is the correct layer: verification requires a key fetch, and doing that per request at your origin is a cache and latency problem you do not need to own.

Cloudflare's implementation verifies signatures at the edge and exposes the result as a request property you can branch on in a rule. Supported operators as of 2026 include Anthropic, OpenAI, Perplexity, Common Crawl, and several Google bots.

Step 3: Verify at the origin, if you must

If you are implementing directly, the flow per request is:

1. Read Signature-Input to learn the key identifier and the covered components.
2. Fetch the operator's key directory over HTTPS. Cache it. A key fetch per inbound request is a self-inflicted denial of service and a hard dependency on someone else's uptime.
3. Verify the Ed25519 signature over the covered components.
4. Check the signature's created and expires values against your clock, with a tolerance measured in minutes, not hours.
5. Confirm the covered components include enough of the request to be meaningful. A signature covering only @authority proves far less than one covering method, path, and authority.

The replay window is the part implementations get wrong. A signature with a long validity and a narrow component set is a bearer token that any observer can reuse against a different path.

Step 4: Fail open, and log it

Treat an unverifiable signature as unverified traffic, not as an attack. Key directories go down, clocks drift, and drafts change. An origin that hard-fails on verification errors will eventually block the exact well-behaved crawlers it was built to admit.

Log four fields on every agent request: the claimed user agent, the verification result, the resolved operator when verification succeeded, and the policy decision. Those four fields are the entire dataset you need to set a real policy in a month.

How this relates to agent identity inside your systems

Web Bot Auth answers "which operator is knocking on my public surface". It says nothing about what an agent may do once it is inside, which is a different problem with different standards.

Confusing the first row with the third is the most common architectural mistake here. A verified signature is an authentication fact about an operator. It is not a grant. See Identity for AI Agents for the authorization side and Secure an MCP Server for the tool boundary.

Should you do it?

You publish content and AI crawlers matter to your traffic

Turn on verification at your CDN, log for a month, then set policy. The measurement is the valuable part and it costs you nothing.

You run an API or paid content

Yes, and treat verified operator identity as a first-class field in your rate-limiting and billing logic. This is the case where "allow named operators only" is a real product decision rather than a defensive crouch.

You are building an agent that fetches the web

Implement signing now. Publish a key directory, sign your requests, and document the operator identity. Unsigned agents are heading toward the same treatment unauthenticated mail gets, and the migration window is closing through late 2026.

You are a small site with no bot problem

Log, do not block. The cost of over-blocking is invisible and the cost of doing nothing is currently zero.

Draft status, operator support, and CDN feature availability all move quickly. Confirm current draft revisions at the IETF datatracker before building against a specific version.