# Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents > How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists. Category: Reference · Updated: 2026-07-07 · Tags: web-bot-auth, http-message-signatures, crawlers, cloudflare, authentication, identity, agents, protocols Canonical: https://changegamer.ai/resources/web-bot-auth Variants: [HTML](https://changegamer.ai/resources/web-bot-auth) · [JSON](https://changegamer.ai/api/resources/web-bot-auth.json) License: https://changegamer.ai/license.xml · Access: free Web Bot Auth is an open mechanism, driven by Cloudflare and built on the IETF's HTTP Message Signatures standard (RFC 9421), that lets a bot or agent cryptographically prove its identity to any origin server — solving the problem that a User-Agent string or IP range (the baseline covered in /resources/ai-crawler-policy) can be spoofed by anyone. If you operate a crawler, browser agent, or autonomous client that repeatedly hits third-party sites, signing your requests is how you get recognized as a legitimate, identifiable agent instead of an anonymous bot. ## Key facts - RFC 9421 (HTTP Message Signatures, IETF) is the underlying signature format. Two companion Cloudflare-authored individual Internet-Drafts define the bot-specific profile — a "directory" draft (how a bot publishes its public keys) and an "architecture" draft (how requests get signed) — neither has advanced past individual-draft status to an IETF working-group RFC as of this writing. - Cloudflare announced Web Bot Auth on May 15, 2025 (closed beta, Ed25519 signatures). OpenAI's Operator agent was already signing its own requests with RFC 9421 at that time. - On July 1, 2025, Cloudflare folded HTTP Message Signature verification into its existing Verified Bots Program, available on Free and Pro plans. - On August 28, 2025, Cloudflare extended the model to "signed agents" — user-directed agents rather than single-company crawlers — naming OpenAI's ChatGPT agent, Block's Goose, Browserbase, and Anchor Browser as early adopters. - As of July 1, 2026 (per Cloudflare's current docs), signed agents are folded into the same "Verified" bot classification, distinguished only by a Direct-vs-Intermediary metadata field. ## Mechanics - Generate an Ed25519 keypair; publish the public key as a signed JWK Set at `/.well-known/http-message-signature-directory` on a domain you control. - Attach three headers to every outbound request: `Signature-Input` (covers `keyid`, `created`/`expires` timestamps, and `tag="web-bot-auth"`), `Signature` (the Ed25519 signature), and `Signature-Agent` (the HTTPS URL of your key directory). - The receiving origin fetches your directory, verifies the signature, and trusts the request's identity — no IP allowlist or self-reported User-Agent required. Cloudflare recommends minute-scale `created`/`expires` windows. ## How this differs from what's already covered robots.txt and User-Agent tokens (/resources/ai-crawler-policy) are declarative and unauthenticated — any client can claim to be `GPTBot`. OAuth/SPIFFE workload identity (/resources/agent-identity-authentication) authenticates an agent to *your own* backend APIs and tools. Web Bot Auth solves a third problem: proving identity to *arbitrary third-party websites* an agent crawls, with no prior relationship or shared secret. ## Checklist - Generate a dedicated Ed25519 key per agent/service; don't reuse across unrelated bots. - Publish a signed key directory at the well-known path; rotate on a schedule. - Sign every request with short-lived `created`/`expires` windows. - Keep sending an honest User-Agent string too (/resources/ai-crawler-policy) — most sites still gate on that layer alone; signing is additive, not yet a full replacement. ## Verified sources - Cloudflare — "Forget IPs: using cryptography to verify bot and agent traffic" (fetched directly, May 15, 2025): https://blog.cloudflare.com/web-bot-auth/ - Cloudflare — "Message Signatures are now part of our Verified Bots Program" (fetched directly, July 1, 2025): https://blog.cloudflare.com/verified-bots-with-cryptography/ - Cloudflare — "The age of agents: cryptographically recognizing agent traffic" (fetched directly, August 28, 2025): https://blog.cloudflare.com/signed-agents/ - Cloudflare Web Bot Auth reference docs (fetched directly): https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/ - Cloudflare Verified Bots concept docs, incl. the July 1, 2026 "signed agents are now Verified" update (fetched directly): https://developers.cloudflare.com/bots/concepts/bot/verified-bots/ - RFC 9421, HTTP Message Signatures (IETF) — cited/quoted by the Cloudflare docs above; not independently fetched (datatracker.ietf.org blocked in this environment): https://datatracker.ietf.org/doc/html/rfc9421 - draft-meunier-web-bot-auth-architecture (IETF individual Internet-Draft, signing protocol) — existence/title corroborated via search only; not independently fetched: https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/ - draft-meunier-http-message-signatures-directory (IETF individual Internet-Draft, key-directory format) — existence/title corroborated via search only; not independently fetched: https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/ - See also: /resources/ai-crawler-policy, /resources/agent-identity-authentication, /resources/agentic-security-checklist --- ## Related resources - [MCP Server Authentication: OAuth 2.1 for Remote Servers](https://changegamer.ai/resources/mcp-server-authentication.md): How OAuth 2.1 works for remote MCP servers: transport differences, Protected Resource Metadata discovery, PKCE, Resource Indicators, and token-audience security — with a step-by-step client flow and honest notes on what ChangeGamer's own /mcp endpoint does. - [Agent Identity and Authentication](https://changegamer.ai/resources/agent-identity-authentication.md): How autonomous agents prove who they are and get authorized to act: workload identity vs. delegated authority, SPIFFE/SPIRE, cloud workload federation, OAuth token exchange, audience binding, and emerging standards — with practical guidance and verified sources. - [Agent Delegation Chains: Credential Propagation in Multi-Agent Systems](https://changegamer.ai/resources/agent-delegation-chains.md): How credential authority flows when one agent spawns another — the multi-hop delegation problem, RFC 8693 token exchange, the act/may_act claims, audience binding across hops, and the IETF drafts standardizing verifiable actor chains in 2026. - [AI Crawler Policy: robots.txt and User-Agents](https://changegamer.ai/resources/ai-crawler-policy.md): Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge. --- Index of all resources: https://changegamer.ai/llms.txt · Full corpus: https://changegamer.ai/llms-full.txt · Corpus data (NDJSON): https://changegamer.ai/api/corpus.jsonl · Offers: https://changegamer.ai/api/pricing.json