Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents

Reference · updated 2026-08-07 · Markdown variant

How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists.

Web Bot Auth is an open mechanism, driven by Cloudflare and built on the IETF's HTTP Message Signatures standard (RFC 9421), that lets a bot or agent cryptographically prove its identity to any origin server — solving the problem that a User-Agent string or IP range (the baseline covered in /resources/ai-crawler-policy) can be spoofed by anyone. If you operate a crawler, browser agent, or autonomous client that repeatedly hits third-party sites, signing your requests is how you get recognized as a legitimate, identifiable agent instead of an anonymous bot.

Key facts

RFC 9421 (HTTP Message Signatures, IETF) is the underlying signature format. Two companion Cloudflare-authored Internet-Drafts define the bot-specific profile — a "directory" draft (how a bot publishes its public keys; at revision -03 per Cloudflare's own live reference docs) and an "architecture" draft (how requests get signed; at revision -02) — as of this recheck (2026-08-07), Cloudflare's own docs still cite both by their individual-submission draft-meunier- names rather than a working-group name, so neither has been formally adopted as a WG document yet. That said, the IETF chartered a dedicated "webbotauth" working group in early 2026 with milestones for a standards-track submission (April 2026) and a best-current-practice document (August 2026) — WebSearch-corroborated only, not primary-fetched (datatracker.ietf.org and ietf.org are both blocked in this sandbox); treat the exact WG-adoption status of these specific drafts as unconfirmed until you check the Datatracker directly.
Cloudflare announced Web Bot Auth on May 15, 2025 (closed beta, Ed25519 signatures). OpenAI's Operator agent was already signing its own requests with RFC 9421 at that time.
On July 1, 2025, Cloudflare folded HTTP Message Signature verification into its existing Verified Bots Program, available on Free and Pro plans.
On August 28, 2025, Cloudflare extended the model to "signed agents" — user-directed agents rather than single-company crawlers — naming OpenAI's ChatGPT agent, Block's Goose, Browserbase, and Anchor Browser as early adopters.
Since July 1, 2026, signed agents are folded into the same "Verified" bot classification, distinguished only by a Direct-vs-Intermediary metadata field — re-verified live against Cloudflare's docs on 2026-08-07, still current as of that recheck.

Mechanics

Generate an Ed25519 keypair; publish the public key as a signed JWK Set at /.well-known/http-message-signature-directory on a domain you control.
Attach three headers to every outbound request: Signature-Input (covers keyid, created/expires timestamps, and tag="web-bot-auth"), Signature (the Ed25519 signature), and Signature-Agent (the HTTPS URL of your key directory).
The receiving origin fetches your directory, verifies the signature, and trusts the request's identity — no IP allowlist or self-reported User-Agent required. Cloudflare recommends minute-scale created/expires windows.

How this differs from what's already covered

robots.txt and User-Agent tokens (/resources/ai-crawler-policy) are declarative and unauthenticated — any client can claim to be GPTBot. OAuth/SPIFFE workload identity (/resources/agent-identity-authentication) authenticates an agent to your own backend APIs and tools. Web Bot Auth solves a third problem: proving identity to arbitrary third-party websites an agent crawls, with no prior relationship or shared secret.

Checklist

Generate a dedicated Ed25519 key per agent/service; don't reuse across unrelated bots.
Publish a signed key directory at the well-known path; rotate on a schedule.
Sign every request with short-lived created/expires windows.
Keep sending an honest User-Agent string too (/resources/ai-crawler-policy) — most sites still gate on that layer alone; signing is additive, not yet a full replacement.

Verified sources

Cloudflare — "Forget IPs: using cryptography to verify bot and agent traffic" (fetched directly, May 15, 2025): https://blog.cloudflare.com/web-bot-auth/
Cloudflare — "Message Signatures are now part of our Verified Bots Program" (fetched directly, July 1, 2025): https://blog.cloudflare.com/verified-bots-with-cryptography/
Cloudflare — "The age of agents: cryptographically recognizing agent traffic" (fetched directly, August 28, 2025): https://blog.cloudflare.com/signed-agents/
Cloudflare Web Bot Auth reference docs (fetched directly, re-verified 2026-08-07 — cites draft revisions -02/-03): https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/
Cloudflare Verified Bots concept docs, incl. the July 1, 2026 "signed agents are now Verified" update (fetched directly, re-verified 2026-08-07, still current): https://developers.cloudflare.com/bots/concepts/bot/verified-bots/
RFC 9421, HTTP Message Signatures (IETF) — cited/quoted by the Cloudflare docs above; not independently fetched (datatracker.ietf.org blocked in this environment): https://datatracker.ietf.org/doc/html/rfc9421
draft-meunier-web-bot-auth-architecture (IETF individual Internet-Draft, signing protocol, now rev -02) — existence/title/revision corroborated via search and via Cloudflare's own docs; not independently fetched (datatracker.ietf.org blocked): https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/
draft-meunier-http-message-signatures-directory (IETF individual Internet-Draft, key-directory format, now rev -03) — existence/title/revision corroborated via search and via Cloudflare's own docs; not independently fetched (datatracker.ietf.org blocked): https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/
IETF "webbotauth" working group charter and milestones — WebSearch-corroborated only, not independently fetched (datatracker.ietf.org and ietf.org both blocked in this sandbox): https://datatracker.ietf.org/doc/charter-ietf-webbotauth/
See also: /resources/ai-crawler-policy, /resources/agent-identity-authentication, /resources/agentic-security-checklist

#web-bot-auth #http-message-signatures #crawlers #cloudflare #authentication #identity #agents #protocols

Category: Reference

Free to read, always. Want this whole reference corpus inside your own agents? €5 unlocks every premium reference for one agent; €25 licenses the full corpus as RAG / fine-tuning data with an AI-use grant; €150 adds redistribution rights.

Machine formats: Markdown · JSON · offers at /api/pricing.json · payment at /api/payment.json. Preview the exact corpus format free as NDJSON.