← All resources

AI Crawler Policy: robots.txt and User-Agents

Reference · updated 2026-07-07 · Markdown variant

Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.

robots.txt is advisory. A compliant crawler reads it before fetching and honours Disallow rules — but WAF and firewall rules are enforced earlier, at the network edge, before a crawler can even retrieve robots.txt. If your WAF blocks a UA, robots.txt Allow rules have no effect. This page covers both layers.

Key facts

• robots.txt only binds crawlers that choose to honor it; WAF and firewall rules sit further upstream, at the edge of the network, and can block a user-agent before it ever reaches robots.txt, so an Allow rule there doesn't override a WAF block.
• Most vendor training and search crawlers are documented as robots.txt-compliant, but notable exceptions exist: Perplexity says its live-fetch agent ignores robots.txt outright, OpenAI's live-fetch agent dropped off its compliant-agent list per a December 2025 documentation update, and a couple of other vendors' training bots carry disputed or documented non-compliance histories.
• Two tokens are not separate crawlers at all but training-opt-out signals layered onto an existing search crawler's fetches — disallowing them stops training use without touching search indexing.
• Because any client can send a fake user-agent string, cross-checking a request's IP range against the vendor's published list is the only dependable way to know a crawler is genuine.
• ChangeGamer's own operational history found a Cloudflare-managed bot-protection rule silently blocking several compliant crawlers despite robots.txt clearly permitting them — a reminder that a WAF layer can override robots.txt with no visible error.
• The fix that worked was turning off an overly aggressive browser-integrity check, setting the crawler-control feature to allow all crawlers, and adding a WAF rule that skips managed rules for matched crawler user-agents — verified afterward with a spoofed-UA curl expecting success rather than trusting the robots.txt file alone.
• Newer access-control layers building on top of robots.txt include a machine-readable licensing standard (discovered via a header directive, still early in its adoption curve as of June 2026) and HTTP 402 pay-per-crawl gates, including a major CDN vendor's own pay-per-crawl offering that was in a private beta stage that same month.

Canonical user-agent token table

robots.txt syntax — per-UA examples

# Allow ChatGPT Search indexing; block training
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

# Block Anthropic training; allow user fetches and search
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Keep Google Search; block AI training use
User-agent: Google-Extended
Disallow: /

# Block Apple Intelligence training; keep Apple Search
User-agent: Applebot-Extended
Disallow: /

# Block Common Crawl (source data for most LLMs)
User-agent: CCBot
Disallow: /

The advisory-only limit

robots.txt binds only crawlers that voluntarily read and respect it. Practical gaps:

• User-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) are sent by live users who requested a specific URL. Perplexity explicitly states Perplexity-User ignores robots.txt. As of a Dec 2025 documentation update, OpenAI no longer lists ChatGPT-User among its robots.txt-compliant agents (only GPTBot and OAI-SearchBot are) — it is now treated as a proxy for human browsing. Meta does not clearly state whether Meta-ExternalFetcher honours Disallow.
• Non-compliant crawlers (Bytespider, Meta-ExternalAgent) have documented or disputed histories of ignoring Disallow. IP-range blocking and WAF rules provide a harder layer.
• Spoofed UAs — any actor can send a fake UA. Vendor IP-range verification is the only way to confirm a crawler is authentic. All major vendors publish IP ranges (see verified sources below).

Training opt-out tokens vs real crawlers

Two tokens are semantic policy signals, not user agents of separate crawlers:

• Google-Extended — The physical crawler is still Googlebot. Disallowing Google-Extended tells Google not to use already-crawled content to train Gemini and Vertex AI. It does not affect Google Search inclusion or ranking.
• Applebot-Extended — The physical crawler is Applebot. Disallowing Applebot-Extended tells Apple not to use already-crawled content to train Apple Intelligence and foundation models. Apple Search / Spotlight inclusion is unaffected.

The WAF/edge layer: sits above robots.txt

ChangeGamer's own experience (BACKLOG item 0, June 2026): Cloudflare's managed rule "Manage AI bots" (firewallManaged) was silently 403ing GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, CCBot, and Google-CloudVertexBot — including on / and /sitemap.xml — even though robots.txt explicitly allowed them. Cloudflare Browser Integrity Check (BIC, enabled by default) additionally 403'd any client without standard browser headers, breaking the Google Search Console sitemap fetch.

Fixes: BIC off; AI Crawl Control set to Allow for all crawlers; WAF custom rule "Allow AI crawlers" (Skip all managed rules, UA-match, logging on) to ensure managed rules cannot re-block them.

Key lesson: after any WAF or security-rule change, verify actual crawler access with a spoofed-UA curl against your live domain — do not assume robots.txt Allow is sufficient:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  -I https://yourdomain.com/

curl -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://anthropic.com/aup)" \
  -I https://yourdomain.com/

Expect HTTP 200. A 403 means the WAF or BIC is blocking at the edge, not robots.txt.

Emerging access-control and monetization signals

robots.txt is the established baseline, but newer mechanisms layer on top:

• RSL (Really Simple Licensing) — an XML-based machine-readable license document (a standalone .xml file, e.g. /license.xml) that declares licensing terms, usage boundaries, and compensation requirements. It is discovered via a License: directive in robots.txt (and HTTP headers, RSS, or HTML <link>). Spec at rslstandard.org. Announced 2025; early adoption stage as of June 2026. ChangeGamer publishes its own at /license.xml.
• HTTP 402 / pay-per-crawl — direct programmatic payment gate on individual resource requests. See /resources/paying-for-access-402 and /resources/access-and-pricing.
• Cloudflare Pay Per Crawl — Cloudflare's 402-based per-crawl pricing at the CDN layer (private beta as of June 2026). See /resources/access-and-pricing for current status.

For how agents should respond to a 402 gate, see /resources/paying-for-access-402. For how ChangeGamer publishes its own machine-readable content index, see /resources/llms-txt-explained.

Verified sources

• OpenAI crawler overview: https://developers.openai.com/api/docs/bots
• OpenAI publishers FAQ: https://help.openai.com/en/articles/12627856-publishers-and-developers-faq
• Anthropic crawler support page: https://support.anthropic.com/en/articles/8896518
• Google common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Applebot support page: https://support.apple.com/en-us/119829
• Perplexity bots guide: https://docs.perplexity.ai/guides/bots
• Amazon Amazonbot: https://developer.amazon.com/amazonbot
• Common Crawl CCBot: https://commoncrawl.org/ccbot
• Mistral AI robots doc: https://docs.mistral.ai/robots
• Cloudflare AI Crawl Control bot reference: https://developers.cloudflare.com/ai-crawl-control/reference/bots/
• Cloudflare Browser Integrity Check: https://developers.cloudflare.com/waf/tools/browser-integrity-check/
• RSL (Really Simple Licensing): https://rslstandard.org/
• Search Engine Roundtable, OpenAI ChatGPT-User/OAI-SearchBot doc update (Dec 2025): https://www.seroundtable.com/openai-chatgpt-crawler-oai-searchbot-update-40558.html
• DuckDuckGo DuckAssistBot help page: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot

#crawlers #robots.txt #user-agents #cloudflare #access-control

Category: Reference