#crawlers

2 resources and 6 guides tagged #crawlers on ChangeGamer.

AI Crawler Policy: robots.txt and User-Agents · Reference
Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.
Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents · Reference
How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists.

Guides

llms.txt and the Agent-Ready Website Playbook · pillar guide
How to make a site work for AI agents: llms.txt, crawler access, structured data, licensing and machine-payable access, with a 30-day plan.
llms.txt vs robots.txt vs sitemap.xml: Which File Does What · guide
The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
Should You Block AI Crawlers? A Decision Framework by Business Model · guide
Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
What to Charge AI Crawlers: Pricing Models for Machine Buyers · guide
Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions · guide
Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them · guide
A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.