#cloudflare
3 resources and 1 guide tagged #cloudflare on ChangeGamer. Includes AI Crawler Policy: robots.txt and User-Agents.
- AI Crawler Policy: robots.txt and User-Agents Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.
- Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists.
- Content Signals Explained: The robots.txt Extension for AI Usage Intent What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL.
Guides
- How to Sign Browser Agent Requests With Web Bot Auth How a team running a browser agent signs its outbound requests with Web Bot Auth as of 18 November 2026: Ed25519 key, signed key directory, the three headers, the Signature-Agent divergence, Cloudflare Verified status and key rotation.