{
  "slug": "web-bot-auth",
  "title": "Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents",
  "description": "How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists.",
  "category": "Reference",
  "tags": [
    "web-bot-auth",
    "http-message-signatures",
    "crawlers",
    "cloudflare",
    "authentication",
    "identity",
    "agents",
    "protocols"
  ],
  "updated": "2026-07-07",
  "canonical": "https://changegamer.ai/resources/web-bot-auth",
  "markdown": "https://changegamer.ai/resources/web-bot-auth.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "Mechanics",
      "anchor": "mechanics"
    },
    {
      "depth": 2,
      "text": "How this differs from what's already covered",
      "anchor": "how-this-differs-from-what-s-already-covered"
    },
    {
      "depth": 2,
      "text": "Checklist",
      "anchor": "checklist"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "mcp-server-authentication",
      "title": "MCP Server Authentication: OAuth 2.1 for Remote Servers",
      "description": "How OAuth 2.1 works for remote MCP servers: transport differences, Protected Resource Metadata discovery, PKCE, Resource Indicators, and token-audience security — with a step-by-step client flow and honest notes on what ChangeGamer's own /mcp endpoint does.",
      "url": "https://changegamer.ai/resources/mcp-server-authentication"
    },
    {
      "slug": "agent-identity-authentication",
      "title": "Agent Identity and Authentication",
      "description": "How autonomous agents prove who they are and get authorized to act: workload identity vs. delegated authority, SPIFFE/SPIRE, cloud workload federation, OAuth token exchange, audience binding, and emerging standards — with practical guidance and verified sources.",
      "url": "https://changegamer.ai/resources/agent-identity-authentication"
    },
    {
      "slug": "agent-delegation-chains",
      "title": "Agent Delegation Chains: Credential Propagation in Multi-Agent Systems",
      "description": "How credential authority flows when one agent spawns another — the multi-hop delegation problem, RFC 8693 token exchange, the act/may_act claims, audience binding across hops, and the IETF drafts standardizing verifiable actor chains in 2026.",
      "url": "https://changegamer.ai/resources/agent-delegation-chains"
    },
    {
      "slug": "ai-crawler-policy",
      "title": "AI Crawler Policy: robots.txt and User-Agents",
      "description": "Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.",
      "url": "https://changegamer.ai/resources/ai-crawler-policy"
    }
  ],
  "body": "Web Bot Auth is an open mechanism, driven by Cloudflare and built on the IETF's HTTP Message Signatures standard (RFC 9421), that lets a bot or agent cryptographically prove its identity to any origin server — solving the problem that a User-Agent string or IP range (the baseline covered in /resources/ai-crawler-policy) can be spoofed by anyone. If you operate a crawler, browser agent, or autonomous client that repeatedly hits third-party sites, signing your requests is how you get recognized as a legitimate, identifiable agent instead of an anonymous bot.\n\n## Key facts\n\n- RFC 9421 (HTTP Message Signatures, IETF) is the underlying signature format. Two companion Cloudflare-authored individual Internet-Drafts define the bot-specific profile — a \"directory\" draft (how a bot publishes its public keys) and an \"architecture\" draft (how requests get signed) — neither has advanced past individual-draft status to an IETF working-group RFC as of this writing.\n- Cloudflare announced Web Bot Auth on May 15, 2025 (closed beta, Ed25519 signatures). OpenAI's Operator agent was already signing its own requests with RFC 9421 at that time.\n- On July 1, 2025, Cloudflare folded HTTP Message Signature verification into its existing Verified Bots Program, available on Free and Pro plans.\n- On August 28, 2025, Cloudflare extended the model to \"signed agents\" — user-directed agents rather than single-company crawlers — naming OpenAI's ChatGPT agent, Block's Goose, Browserbase, and Anchor Browser as early adopters.\n- As of July 1, 2026 (per Cloudflare's current docs), signed agents are folded into the same \"Verified\" bot classification, distinguished only by a Direct-vs-Intermediary metadata field.\n\n## Mechanics\n\n- Generate an Ed25519 keypair; publish the public key as a signed JWK Set at `/.well-known/http-message-signature-directory` on a domain you control.\n- Attach three headers to every outbound request: `Signature-Input` (covers `keyid`, `created`/`expires` timestamps, and `tag=\"web-bot-auth\"`), `Signature` (the Ed25519 signature), and `Signature-Agent` (the HTTPS URL of your key directory).\n- The receiving origin fetches your directory, verifies the signature, and trusts the request's identity — no IP allowlist or self-reported User-Agent required. Cloudflare recommends minute-scale `created`/`expires` windows.\n\n## How this differs from what's already covered\n\nrobots.txt and User-Agent tokens (/resources/ai-crawler-policy) are declarative and unauthenticated — any client can claim to be `GPTBot`. OAuth/SPIFFE workload identity (/resources/agent-identity-authentication) authenticates an agent to *your own* backend APIs and tools. Web Bot Auth solves a third problem: proving identity to *arbitrary third-party websites* an agent crawls, with no prior relationship or shared secret.\n\n## Checklist\n\n- Generate a dedicated Ed25519 key per agent/service; don't reuse across unrelated bots.\n- Publish a signed key directory at the well-known path; rotate on a schedule.\n- Sign every request with short-lived `created`/`expires` windows.\n- Keep sending an honest User-Agent string too (/resources/ai-crawler-policy) — most sites still gate on that layer alone; signing is additive, not yet a full replacement.\n\n## Verified sources\n\n- Cloudflare — \"Forget IPs: using cryptography to verify bot and agent traffic\" (fetched directly, May 15, 2025): https://blog.cloudflare.com/web-bot-auth/\n- Cloudflare — \"Message Signatures are now part of our Verified Bots Program\" (fetched directly, July 1, 2025): https://blog.cloudflare.com/verified-bots-with-cryptography/\n- Cloudflare — \"The age of agents: cryptographically recognizing agent traffic\" (fetched directly, August 28, 2025): https://blog.cloudflare.com/signed-agents/\n- Cloudflare Web Bot Auth reference docs (fetched directly): https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/\n- Cloudflare Verified Bots concept docs, incl. the July 1, 2026 \"signed agents are now Verified\" update (fetched directly): https://developers.cloudflare.com/bots/concepts/bot/verified-bots/\n- RFC 9421, HTTP Message Signatures (IETF) — cited/quoted by the Cloudflare docs above; not independently fetched (datatracker.ietf.org blocked in this environment): https://datatracker.ietf.org/doc/html/rfc9421\n- draft-meunier-web-bot-auth-architecture (IETF individual Internet-Draft, signing protocol) — existence/title corroborated via search only; not independently fetched: https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/\n- draft-meunier-http-message-signatures-directory (IETF individual Internet-Draft, key-directory format) — existence/title corroborated via search only; not independently fetched: https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/\n- See also: /resources/ai-crawler-policy, /resources/agent-identity-authentication, /resources/agentic-security-checklist",
  "sources": [
    "https://blog.cloudflare.com/web-bot-auth/",
    "https://blog.cloudflare.com/verified-bots-with-cryptography/",
    "https://blog.cloudflare.com/signed-agents/",
    "https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/",
    "https://developers.cloudflare.com/bots/concepts/bot/verified-bots/",
    "https://datatracker.ietf.org/doc/html/rfc9421",
    "https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/",
    "https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/"
  ]
}