Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents
How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists.
Web Bot Auth is an open mechanism, driven by Cloudflare and built on the IETF's HTTP Message Signatures standard (RFC 9421), that lets a bot or agent cryptographically prove its identity to any origin server — solving the problem that a User-Agent string or IP range (the baseline covered in /resources/ai-crawler-policy) can be spoofed by anyone. If you operate a crawler, browser agent, or autonomous client that repeatedly hits third-party sites, signing your requests is how you get recognized as a legitimate, identifiable agent instead of an anonymous bot.
Key facts
- RFC 9421 (HTTP Message Signatures, IETF) is the underlying signature format. Two companion Cloudflare-authored individual Internet-Drafts define the bot-specific profile — a "directory" draft (how a bot publishes its public keys) and an "architecture" draft (how requests get signed) — neither has advanced past individual-draft status to an IETF working-group RFC as of this writing.
- Cloudflare announced Web Bot Auth on May 15, 2025 (closed beta, Ed25519 signatures). OpenAI's Operator agent was already signing its own requests with RFC 9421 at that time.
- On July 1, 2025, Cloudflare folded HTTP Message Signature verification into its existing Verified Bots Program, available on Free and Pro plans.
- On August 28, 2025, Cloudflare extended the model to "signed agents" — user-directed agents rather than single-company crawlers — naming OpenAI's ChatGPT agent, Block's Goose, Browserbase, and Anchor Browser as early adopters.
- As of July 1, 2026 (per Cloudflare's current docs), signed agents are folded into the same "Verified" bot classification, distinguished only by a Direct-vs-Intermediary metadata field.
Mechanics
- Generate an Ed25519 keypair; publish the public key as a signed JWK Set at
/.well-known/http-message-signature-directoryon a domain you control. - Attach three headers to every outbound request:
Signature-Input(coverskeyid,created/expirestimestamps, andtag="web-bot-auth"),Signature(the Ed25519 signature), andSignature-Agent(the HTTPS URL of your key directory). - The receiving origin fetches your directory, verifies the signature, and trusts the request's identity — no IP allowlist or self-reported User-Agent required. Cloudflare recommends minute-scale
created/expireswindows.
How this differs from what's already covered
robots.txt and User-Agent tokens (/resources/ai-crawler-policy) are declarative and unauthenticated — any client can claim to be GPTBot. OAuth/SPIFFE workload identity (/resources/agent-identity-authentication) authenticates an agent to your own backend APIs and tools. Web Bot Auth solves a third problem: proving identity to arbitrary third-party websites an agent crawls, with no prior relationship or shared secret.
Checklist
- Generate a dedicated Ed25519 key per agent/service; don't reuse across unrelated bots.
- Publish a signed key directory at the well-known path; rotate on a schedule.
- Sign every request with short-lived
created/expireswindows. - Keep sending an honest User-Agent string too (/resources/ai-crawler-policy) — most sites still gate on that layer alone; signing is additive, not yet a full replacement.
Verified sources
- Cloudflare — "Forget IPs: using cryptography to verify bot and agent traffic" (fetched directly, May 15, 2025): https://blog.cloudflare.com/web-bot-auth/
- Cloudflare — "Message Signatures are now part of our Verified Bots Program" (fetched directly, July 1, 2025): https://blog.cloudflare.com/verified-bots-with-cryptography/
- Cloudflare — "The age of agents: cryptographically recognizing agent traffic" (fetched directly, August 28, 2025): https://blog.cloudflare.com/signed-agents/
- Cloudflare Web Bot Auth reference docs (fetched directly): https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/
- Cloudflare Verified Bots concept docs, incl. the July 1, 2026 "signed agents are now Verified" update (fetched directly): https://developers.cloudflare.com/bots/concepts/bot/verified-bots/
- RFC 9421, HTTP Message Signatures (IETF) — cited/quoted by the Cloudflare docs above; not independently fetched (datatracker.ietf.org blocked in this environment): https://datatracker.ietf.org/doc/html/rfc9421
- draft-meunier-web-bot-auth-architecture (IETF individual Internet-Draft, signing protocol) — existence/title corroborated via search only; not independently fetched: https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/
- draft-meunier-http-message-signatures-directory (IETF individual Internet-Draft, key-directory format) — existence/title corroborated via search only; not independently fetched: https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/
- See also: /resources/ai-crawler-policy, /resources/agent-identity-authentication, /resources/agentic-security-checklist