{
  "slug": "ai-crawler-policy",
  "title": "AI Crawler Policy: robots.txt and User-Agents",
  "description": "Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.",
  "category": "Reference",
  "tags": [
    "crawlers",
    "robots.txt",
    "user-agents",
    "cloudflare",
    "access-control"
  ],
  "updated": "2026-07-07",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/ai-crawler-policy",
  "markdown": "https://changegamer.ai/resources/ai-crawler-policy.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "Canonical user-agent token table",
      "anchor": "canonical-user-agent-token-table"
    },
    {
      "depth": 2,
      "text": "robots.txt syntax — per-UA examples",
      "anchor": "robots-txt-syntax-per-ua-examples"
    },
    {
      "depth": 2,
      "text": "The advisory-only limit",
      "anchor": "the-advisory-only-limit"
    },
    {
      "depth": 2,
      "text": "Training opt-out tokens vs real crawlers",
      "anchor": "training-opt-out-tokens-vs-real-crawlers"
    },
    {
      "depth": 2,
      "text": "The WAF/edge layer: sits above robots.txt",
      "anchor": "the-waf-edge-layer-sits-above-robots-txt"
    },
    {
      "depth": 2,
      "text": "Emerging access-control and monetization signals",
      "anchor": "emerging-access-control-and-monetization-signals"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "content-signals-explained",
      "title": "Content Signals Explained: The robots.txt Extension for AI Usage Intent",
      "description": "What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL.",
      "url": "https://changegamer.ai/resources/content-signals-explained"
    },
    {
      "slug": "web-bot-auth",
      "title": "Web Bot Auth: Cryptographically Verifying AI Crawlers and Agents",
      "description": "How Web Bot Auth — Cloudflare's implementation of IETF HTTP Message Signatures (RFC 9421) — lets a crawler or agent cryptographically prove its identity to a website, replacing the spoofable User-Agent string and brittle IP allowlists.",
      "url": "https://changegamer.ai/resources/web-bot-auth"
    },
    {
      "slug": "agent-control-specification",
      "title": "Agent Control Specification (ACS): Portable Runtime Policy Enforcement for Agents",
      "description": "Microsoft's open, MIT-licensed specification for enforcing policy at defined checkpoints in an agent's execution loop — the eight intervention points, five verdict types, and how it differs from guardrail tooling and other agent-security specs already on ChangeGamer.",
      "url": "https://changegamer.ai/resources/agent-control-specification"
    },
    {
      "slug": "agent-delegation-chains",
      "title": "Agent Delegation Chains: Credential Propagation in Multi-Agent Systems",
      "description": "How credential authority flows when one agent spawns another — the multi-hop delegation problem, RFC 8693 token exchange, the act/may_act claims, audience binding across hops, and the IETF drafts standardizing verifiable actor chains in 2026.",
      "url": "https://changegamer.ai/resources/agent-delegation-chains"
    }
  ],
  "furtherReading": [
    {
      "slug": "llms-txt-vs-robots-txt-vs-sitemap",
      "title": "llms.txt vs robots.txt vs sitemap.xml: Which File Does What",
      "description": "The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.",
      "url": "https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap"
    },
    {
      "slug": "should-you-block-ai-crawlers",
      "title": "Should You Block AI Crawlers? A Decision Framework by Business Model",
      "description": "Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.",
      "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers"
    }
  ],
  "body": "robots.txt is advisory. A compliant crawler reads it before fetching and honours Disallow rules — but WAF and firewall rules are enforced earlier, at the network edge, before a crawler can even retrieve robots.txt. If your WAF blocks a UA, robots.txt Allow rules have no effect. This page covers both layers.\n\n## Key facts\n\n- robots.txt only binds crawlers that choose to honor it; WAF and firewall rules sit further upstream, at the edge of the network, and can block a user-agent before it ever reaches robots.txt, so an Allow rule there doesn't override a WAF block.\n- Most vendor training and search crawlers are documented as robots.txt-compliant, but notable exceptions exist: Perplexity says its live-fetch agent ignores robots.txt outright, OpenAI's live-fetch agent dropped off its compliant-agent list per a December 2025 documentation update, and a couple of other vendors' training bots carry disputed or documented non-compliance histories.\n- Two tokens are not separate crawlers at all but training-opt-out signals layered onto an existing search crawler's fetches — disallowing them stops training use without touching search indexing.\n- Because any client can send a fake user-agent string, cross-checking a request's IP range against the vendor's published list is the only dependable way to know a crawler is genuine.\n- ChangeGamer's own operational history found a Cloudflare-managed bot-protection rule silently blocking several compliant crawlers despite robots.txt clearly permitting them — a reminder that a WAF layer can override robots.txt with no visible error.\n- The fix that worked was turning off an overly aggressive browser-integrity check, setting the crawler-control feature to allow all crawlers, and adding a WAF rule that skips managed rules for matched crawler user-agents — verified afterward with a spoofed-UA curl expecting success rather than trusting the robots.txt file alone.\n- Newer access-control layers building on top of robots.txt include a machine-readable licensing standard (discovered via a header directive, still early in its adoption curve as of June 2026) and HTTP 402 pay-per-crawl gates, including a major CDN vendor's own pay-per-crawl offering that was in a private beta stage that same month.\n\n## Canonical user-agent token table\n\n| Token | Vendor | Purpose | robots.txt honoured? |\n|---|---|---|---|\n| `GPTBot` | OpenAI | Training data collection for GPT models | Yes |\n| `OAI-SearchBot` | OpenAI | Indexing for ChatGPT Search (not training) | Yes |\n| `ChatGPT-User` | OpenAI | Live user-triggered page fetch, incl. Custom GPT / GPT Actions requests | No — a Dec 2025 OpenAI doc update lists only `GPTBot`/`OAI-SearchBot` as robots.txt-compliant |\n| `ClaudeBot` | Anthropic | Training data collection for Claude models | Yes |\n| `Claude-SearchBot` | Anthropic | Indexing for Claude search results | Yes |\n| `Claude-User` | Anthropic | Live user-triggered page fetch | Yes (Anthropic states all three honour it) |\n| `Google-Extended` | Google | Training opt-out token for Gemini/Vertex AI — NOT a separate crawler; Googlebot fetches, this token controls downstream use | Yes (training opt-out only) |\n| `Googlebot` | Google | Google Search indexing; also executes Google-Extended policy | Yes |\n| `Google-CloudVertexBot` | Google | Crawls at site-owner request during Vertex AI Agent development | Yes |\n| `PerplexityBot` | Perplexity | Indexing for Perplexity search answers | Yes |\n| `Perplexity-User` | Perplexity | Live user-triggered page fetch | No — ignores robots.txt by design |\n| `DuckAssistBot` | DuckDuckGo | Live user-triggered fetch for DuckAssist AI answers (RAG); not used for training | Yes — Disallow opt-out takes effect ~72h later |\n| `Amazonbot` | Amazon | Crawling for Amazon product/AI improvement | Yes |\n| `Applebot` | Apple | Apple Search (Spotlight, Siri) indexing | Yes |\n| `Applebot-Extended` | Apple | Training opt-out token for Apple Intelligence / foundation models — NOT a separate crawler; Applebot fetches, this token controls training use | Yes (training opt-out only) |\n| `Bytespider` | ByteDance | AI training data collection (Doubao LLM) | Disputed — documented violations |\n| `CCBot` | Common Crawl | Open web archive used to train most major LLMs | Yes |\n| `Meta-ExternalAgent` | Meta | Training data for Llama models and Meta AI products (launched July 2024) | Stated yes; compliance disputed |\n| `Meta-ExternalFetcher` | Meta | Live user-triggered fetch for Meta AI assistant answers (parallel to ChatGPT-User/Claude-User); not used for training | Unclear — no independent robots.txt-compliance confirmation found |\n| `MistralAI-User` | Mistral | User-triggered fetch in Le Chat; not used for training | Yes |\n\n## robots.txt syntax — per-UA examples\n\n```\n# Allow ChatGPT Search indexing; block training\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\n# Block Anthropic training; allow user fetches and search\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-SearchBot\nAllow: /\n\nUser-agent: Claude-User\nAllow: /\n\n# Keep Google Search; block AI training use\nUser-agent: Google-Extended\nDisallow: /\n\n# Block Apple Intelligence training; keep Apple Search\nUser-agent: Applebot-Extended\nDisallow: /\n\n# Block Common Crawl (source data for most LLMs)\nUser-agent: CCBot\nDisallow: /\n```\n\n## The advisory-only limit\n\nrobots.txt binds only crawlers that voluntarily read and respect it. Practical gaps:\n\n- **User-triggered fetchers** (`ChatGPT-User`, `Perplexity-User`, `Meta-ExternalFetcher`) are sent by live users who requested a specific URL. Perplexity explicitly states `Perplexity-User` ignores robots.txt. As of a Dec 2025 documentation update, OpenAI no longer lists `ChatGPT-User` among its robots.txt-compliant agents (only `GPTBot` and `OAI-SearchBot` are) — it is now treated as a proxy for human browsing. Meta does not clearly state whether `Meta-ExternalFetcher` honours Disallow.\n- **Non-compliant crawlers** (`Bytespider`, `Meta-ExternalAgent`) have documented or disputed histories of ignoring Disallow. IP-range blocking and WAF rules provide a harder layer.\n- **Spoofed UAs** — any actor can send a fake UA. Vendor IP-range verification is the only way to confirm a crawler is authentic. All major vendors publish IP ranges (see verified sources below).\n\n## Training opt-out tokens vs real crawlers\n\nTwo tokens are semantic policy signals, not user agents of separate crawlers:\n\n- **`Google-Extended`** — The physical crawler is still `Googlebot`. Disallowing `Google-Extended` tells Google not to use already-crawled content to train Gemini and Vertex AI. It does not affect Google Search inclusion or ranking.\n- **`Applebot-Extended`** — The physical crawler is `Applebot`. Disallowing `Applebot-Extended` tells Apple not to use already-crawled content to train Apple Intelligence and foundation models. Apple Search / Spotlight inclusion is unaffected.\n\n## The WAF/edge layer: sits above robots.txt\n\nChangeGamer's own experience (BACKLOG item 0, June 2026): Cloudflare's managed rule \"Manage AI bots\" (`firewallManaged`) was silently 403ing `GPTBot`, `ChatGPT-User`, `OAI-SearchBot`, `PerplexityBot`, `CCBot`, and `Google-CloudVertexBot` — including on `/` and `/sitemap.xml` — even though robots.txt explicitly allowed them. Cloudflare Browser Integrity Check (BIC, enabled by default) additionally 403'd any client without standard browser headers, breaking the Google Search Console sitemap fetch.\n\nFixes: BIC off; AI Crawl Control set to Allow for all crawlers; WAF custom rule \"Allow AI crawlers\" (Skip all managed rules, UA-match, logging on) to ensure managed rules cannot re-block them.\n\n**Key lesson:** after any WAF or security-rule change, verify actual crawler access with a spoofed-UA curl against your live domain — do not assume robots.txt Allow is sufficient:\n\n```bash\ncurl -A \"Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)\" \\\n  -I https://yourdomain.com/\n\ncurl -A \"Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://anthropic.com/aup)\" \\\n  -I https://yourdomain.com/\n```\n\nExpect HTTP 200. A 403 means the WAF or BIC is blocking at the edge, not robots.txt.\n\n## Emerging access-control and monetization signals\n\nrobots.txt is the established baseline, but newer mechanisms layer on top:\n\n- **RSL (Really Simple Licensing)** — an XML-based machine-readable license document (a standalone `.xml` file, e.g. `/license.xml`) that declares licensing terms, usage boundaries, and compensation requirements. It is discovered via a `License:` directive in robots.txt (and HTTP headers, RSS, or HTML `<link>`). Spec at `rslstandard.org`. Announced 2025; early adoption stage as of June 2026. ChangeGamer publishes its own at /license.xml.\n- **HTTP 402 / pay-per-crawl** — direct programmatic payment gate on individual resource requests. See /resources/paying-for-access-402 and /resources/access-and-pricing.\n- **Cloudflare Pay Per Crawl** — Cloudflare's 402-based per-crawl pricing at the CDN layer (private beta as of June 2026). See /resources/access-and-pricing for current status.\n\nFor how agents should respond to a 402 gate, see /resources/paying-for-access-402.\nFor how ChangeGamer publishes its own machine-readable content index, see /resources/llms-txt-explained.\n\n## Verified sources\n\n- OpenAI crawler overview: https://developers.openai.com/api/docs/bots\n- OpenAI publishers FAQ: https://help.openai.com/en/articles/12627856-publishers-and-developers-faq\n- Anthropic crawler support page: https://support.anthropic.com/en/articles/8896518\n- Google common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers\n- Applebot support page: https://support.apple.com/en-us/119829\n- Perplexity bots guide: https://docs.perplexity.ai/guides/bots\n- Amazon Amazonbot: https://developer.amazon.com/amazonbot\n- Common Crawl CCBot: https://commoncrawl.org/ccbot\n- Mistral AI robots doc: https://docs.mistral.ai/robots\n- Cloudflare AI Crawl Control bot reference: https://developers.cloudflare.com/ai-crawl-control/reference/bots/\n- Cloudflare Browser Integrity Check: https://developers.cloudflare.com/waf/tools/browser-integrity-check/\n- RSL (Really Simple Licensing): https://rslstandard.org/\n- Search Engine Roundtable, OpenAI ChatGPT-User/OAI-SearchBot doc update (Dec 2025): https://www.seroundtable.com/openai-chatgpt-crawler-oai-searchbot-update-40558.html\n- DuckDuckGo DuckAssistBot help page: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot",
  "sources": [
    "https://developers.openai.com/api/docs/bots",
    "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq",
    "https://support.anthropic.com/en/articles/8896518",
    "https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers",
    "https://support.apple.com/en-us/119829",
    "https://docs.perplexity.ai/guides/bots",
    "https://developer.amazon.com/amazonbot",
    "https://commoncrawl.org/ccbot",
    "https://docs.mistral.ai/robots",
    "https://developers.cloudflare.com/ai-crawl-control/reference/bots/",
    "https://developers.cloudflare.com/waf/tools/browser-integrity-check/",
    "https://rslstandard.org/",
    "https://www.seroundtable.com/openai-chatgpt-crawler-oai-searchbot-update-40558.html",
    "https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot"
  ]
}