{ "slug": "should-you-block-ai-crawlers", "title": "Should You Block AI Crawlers? A Decision Framework by Business Model", "description": "Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.", "kind": "sub", "order": 5, "target_query": "should I block AI crawlers", "secondary_queries": [ "block GPTBot pros and cons", "does blocking AI crawlers hurt SEO", "allow or block ClaudeBot PerplexityBot" ], "tags": [ "crawlers", "robots.txt", "policy", "monetization", "ai-visibility" ], "published": "2026-07-26", "updated": "2026-07-26", "words": 1175, "premium": false, "license": "https://changegamer.ai/license.xml", "canonical": "https://changegamer.ai/articles/should-you-block-ai-crawlers", "markdown": "https://changegamer.ai/articles/should-you-block-ai-crawlers.md", "takeaways": [ "There is no single \"AI crawler\" decision. There are four: training collection, search indexing, live user-triggered fetches, and open web archives. They pay you back completely differently.", "Training opt-out tokens (`Google-Extended`, `Applebot-Extended`) cost you nothing in search visibility — they are policy tokens, not crawlers.", "Blocking search-indexing bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) removes you from those products' answers. That is a distribution decision, and usually the expensive one to get wrong.", "Blocking is enforcement-free: `robots.txt` is advisory, and some fetchers ignore it by design. If you need enforcement, that is a WAF rule or a 402, not a `Disallow`.", "The middle path most publishers actually want: allow answer-engine indexing, decline training, declare usage preferences, and price bulk access." ], "outline": [ { "depth": 2, "text": "Step 1 — Split \"AI crawler\" into four classes", "anchor": "step-1-split-ai-crawler-into-four-classes", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#step-1-split-ai-crawler-into-four-classes" }, { "depth": 2, "text": "Step 2 — Answer four questions, not one", "anchor": "step-2-answer-four-questions-not-one", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#step-2-answer-four-questions-not-one" }, { "depth": 2, "text": "Step 3 — Read the answer off your business model", "anchor": "step-3-read-the-answer-off-your-business-model", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#step-3-read-the-answer-off-your-business-model" }, { "depth": 3, "text": "Content is the product (news, research, paid archives, data)", "anchor": "content-is-the-product-news-research-paid-archives-data", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#content-is-the-product-news-research-paid-archives-data" }, { "depth": 3, "text": "Content is marketing (SaaS, agencies, tools, most B2B)", "anchor": "content-is-marketing-saas-agencies-tools-most-b2b", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#content-is-marketing-saas-agencies-tools-most-b2b" }, { "depth": 3, "text": "Content is a community asset (forums, UGC, wikis)", "anchor": "content-is-a-community-asset-forums-ugc-wikis", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#content-is-a-community-asset-forums-ugc-wikis" }, { "depth": 3, "text": "Content is regulated, confidential or personal", "anchor": "content-is-regulated-confidential-or-personal", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#content-is-regulated-confidential-or-personal" }, { "depth": 2, "text": "Step 4 — Understand what you have and have not achieved", "anchor": "step-4-understand-what-you-have-and-have-not-achieved", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#step-4-understand-what-you-have-and-have-not-achieved" }, { "depth": 2, "text": "Step 5 — Express usage preferences, not just access", "anchor": "step-5-express-usage-preferences-not-just-access", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#step-5-express-usage-preferences-not-just-access" }, { "depth": 2, "text": "Step 6 — Verify, because policy changes break access", "anchor": "step-6-verify-because-policy-changes-break-access", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#step-6-verify-because-policy-changes-break-access" }, { "depth": 2, "text": "The framework in one paragraph", "anchor": "the-framework-in-one-paragraph", "url": "https://changegamer.ai/articles/should-you-block-ai-crawlers#the-framework-in-one-paragraph" } ], "faq": [ { "question": "Will blocking GPTBot hurt my Google rankings?", "answer": "No. `GPTBot` is OpenAI's training crawler and has no relationship to Google Search. The token that matters for Google is `Googlebot` (search) versus `Google-Extended` (a training opt-out that does not affect search inclusion or ranking)." }, { "question": "If I block training crawlers, is my content still used for training?", "answer": "Possibly. `robots.txt` is advisory; compliance varies, some crawlers have documented histories of ignoring it, and content already collected before you blocked is not withdrawn. Blocking is a forward-looking preference, not a retraction. Content that circulates through third-party mirrors and aggregators is outside your robots.txt entirely." }, { "question": "What is the difference between blocking a crawler and declaring Content Signals?", "answer": "Blocking says \"do not fetch this\". Content Signals say \"you may fetch it, but here is what you may use it for\" — `search`, `ai-input`, `ai-train`, each yes or no. Both are voluntary-compliance mechanisms; signals let you stay visible in answers while declining training use, which blocking cannot express." }, { "question": "Is there a way to actually enforce this?", "answer": "Yes, at two layers: WAF/IP rules stop the request, and HTTP 402 prices it. Everything in robots.txt is a request for cooperation. Vendor IP-range verification matters here too, because user agents are trivially spoofed." } ], "body": "\"Should we block AI crawlers?\" is the wrong question, and asking it that way is why so many sites end up with a `robots.txt` that costs them distribution without protecting anything. There are four decisions hiding inside it, and they have different answers for almost every business.\n\nThis is the control chapter of [the agent-ready website](/articles/agent-ready-website). The maintained token reference behind it is [AI crawler policy](/resources/ai-crawler-policy).\n\n## Step 1 — Split \"AI crawler\" into four classes\n\n| Class | Example tokens | What it does for you | What it takes |\n|---|---|---|---|\n| **Training collection** | `GPTBot`, `ClaudeBot`, `Meta-ExternalAgent`, `Bytespider` | Nothing directly — no traffic, no citation | Your content becomes model weights |\n| **Search / answer indexing** | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `DuckAssistBot`, `Googlebot` | Presence in answers, some referral clicks | A fetch; sometimes answers replace the click |\n| **Live user fetch** | `ChatGPT-User`, `Perplexity-User`, `Claude-User`, `Meta-ExternalFetcher`, `MistralAI-User` | A real person asked for your page, right now | Almost nothing — this is closest to a human visit |\n| **Open archive** | `CCBot` (Common Crawl) | Indirect reach; research and downstream use | Your content feeds many models at once |\n\nAdd a fifth row that is not a crawler at all: **training opt-out tokens** — `Google-Extended` and `Applebot-Extended`. The fetch is still done by `Googlebot` and `Applebot`; the token controls downstream training use only. This is the cheapest decision on the page: disallowing them costs no search visibility.\n\n## Step 2 — Answer four questions, not one\n\n**Q1. Do you want to be *in* AI answers?** For nearly every business whose customers ask questions before buying, yes. Answer engines are a discovery surface; being absent means a competitor is cited instead. Blocking search-indexing bots is the decision most likely to be regretted.\n\n**Q2. Do you want your content used as training data?** This is where content-as-product businesses diverge from everyone else. If your archive is your asset, declining training use is coherent. If your content is marketing for something else you sell, training exposure is closer to free brand distribution than to theft.\n\n**Q3. Do you want live, user-triggered fetches?** Almost always yes. A person asked for your page. Blocking this class is closest to blocking a browser.\n\n**Q4. Do you want bulk archival collection?** Common Crawl is one fetch that feeds many downstream models and much academic work. If you decline training, blocking `CCBot` is the consistent follow-through; if you have not decided, this is a low-stakes decision.\n\n## Step 3 — Read the answer off your business model\n\n### Content is the product (news, research, paid archives, data)\n\nAllow answer indexing and live fetches, decline training, and price bulk access. Blanket blocking removes you from discovery while doing nothing about content already collected, and it forgoes the only revenue path that actually exists.\n\n```\nUser-agent: OAI-SearchBot\nAllow: /\nUser-agent: Claude-SearchBot\nAllow: /\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\nUser-agent: ClaudeBot\nDisallow: /\nUser-agent: CCBot\nDisallow: /\nUser-agent: Google-Extended\nDisallow: /\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: *\nContent-Signal: search=yes, ai-train=no\n\nSitemap: https://example.com/sitemap.xml\nLicense: https://example.com/license.xml\n```\n\nThen put a price on the bulk path so the answer to \"can I license this?\" is machine-readable rather than a contact form: [what to charge AI crawlers](/articles/what-to-charge-ai-crawlers) and [licensing content for AI training](/articles/licensing-content-for-ai-training).\n\n### Content is marketing (SaaS, agencies, tools, most B2B)\n\nAllow essentially everything. Your docs being inside a model that developers ask for help is a distribution win, not a leak. The realistic risk is not theft, it is being *absent* from the answer when someone asks which tool solves their problem.\n\n```\nUser-agent: *\nAllow: /\nContent-Signal: search=yes, ai-input=yes\n\nSitemap: https://example.com/sitemap.xml\n```\n\n### Content is a community asset (forums, UGC, wikis)\n\nThe decision is not really yours alone — contributors have expectations, and your terms may constrain what you can license. Declining training while allowing answer indexing is the defensible default, and the reasoning belongs in public.\n\n### Content is regulated, confidential or personal\n\nDo not rely on `robots.txt` at all. Advisory files are the wrong tool for material that must not be ingested: use authentication, and note that a training crawler with valid credentials is not a scenario robots.txt addresses. Data-handling obligations do not disappear because the client is a model.\n\n## Step 4 — Understand what you have and have not achieved\n\nThree limits, all of which matter more than the syntax:\n\n1. **It is advisory.** Compliance is voluntary. Perplexity states `Perplexity-User` ignores robots.txt by design; since a December 2025 documentation update OpenAI no longer lists `ChatGPT-User` among its robots.txt-compliant agents; some crawlers have documented histories of ignoring `Disallow`.\n2. **It is forward-looking only.** A `Disallow` added today (as of July 2026) does not retract content collected last year, nor content mirrored on sites you do not control.\n3. **User agents are spoofable.** Anyone can claim to be `GPTBot`. Vendor IP-range verification is the only way to know, and every major vendor publishes ranges — see the source list in [AI crawler policy](/resources/ai-crawler-policy).\n\nIf any of those limits is unacceptable for your content, the honest options are enforcement (WAF/IP rules) or pricing (HTTP 402) — see [implementing an HTTP 402 paywall](/articles/http-402-paywall-implementation) and [agentic payment protocols](/resources/agentic-payment-protocols).\n\n## Step 5 — Express usage preferences, not just access\n\nBlocking is a blunt instrument: it cannot say \"index me but do not train on me\". Content Signals can. Cloudflare announced the Content Signals Policy on 24 September 2025 with three directives — `search`, `ai-input` (real-time generative use such as grounding) and `ai-train` — and defaulted it on for over 3.8 million domains at `search=yes, ai-train=no`. Cloudflare is explicit that these are preferences rather than a technical block, while noting they \"might have legal rights in various jurisdictions\", citing the EU text-and-data-mining opt-out in Directive 2019/790. The IETF's AIPREF Working Group, chartered in 2025, is standardising the same idea; as of mid-2026 it is still in progress, not an RFC. Details: [content signals explained](/resources/content-signals-explained).\n\n## Step 6 — Verify, because policy changes break access\n\nEvery robots.txt edit deserves the same follow-up, since the layer that actually decides is your edge:\n\n```bash\nfor ua in \"GPTBot/1.2\" \"OAI-SearchBot/1.0\" \"ClaudeBot/1.0\" \"PerplexityBot/1.0\"; do\n printf \"%-22s \" \"$ua\"\n curl -sI -A \"$ua\" https://example.com/ | head -1\ndone\n```\n\nA `403` where you intended an `Allow` means bot management is overriding your published policy — the exact failure this site hit when Cloudflare's managed AI-bots rule silently 403'd six crawlers that `robots.txt` explicitly allowed. Everything that can go wrong here is catalogued in [why AI agents can't read your site](/articles/why-ai-agents-cant-read-your-site).\n\n## The framework in one paragraph\n\nAllow live user fetches always. Allow answer indexing unless you have a specific reason not to be discoverable. Decide training separately, and express that decision with both a `Disallow` and a Content Signal, because they say different things. Publish a licence if your content has licensing value, and a price if you want revenue rather than a preference. Then verify at the edge — because a policy your CDN overrides is not a policy, it is a document.", "cluster": { "id": "agent-ready-web", "title": "The agent-ready web", "description": "How to make a website readable, citable, controllable and payable for AI agents and AI crawlers — the operator side of the machine-first web.", "status": "complete", "pillar": { "slug": "agent-ready-website", "title": "The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization", "description": "The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.", "kind": "pillar", "order": 0, "html": "https://changegamer.ai/articles/agent-ready-website", "markdown": "https://changegamer.ai/articles/agent-ready-website.md", "json": "https://changegamer.ai/api/articles/agent-ready-website.json" }, "articles": [ { "slug": "llms-txt-vs-robots-txt-vs-sitemap", "title": "llms.txt vs robots.txt vs sitemap.xml: Which File Does What", "description": "The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.", "kind": "sub", "order": 1, "html": "https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap", "markdown": "https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md", "json": "https://changegamer.ai/api/articles/llms-txt-vs-robots-txt-vs-sitemap.json" }, { "slug": "how-to-write-an-llms-txt-file", "title": "How to Write an llms.txt File (Format, Template, and Maintenance)", "description": "A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.", "kind": "sub", "order": 2, "html": "https://changegamer.ai/articles/how-to-write-an-llms-txt-file", "markdown": "https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md", "json": "https://changegamer.ai/api/articles/how-to-write-an-llms-txt-file.json" }, { "slug": "serving-markdown-variants-to-ai-agents", "title": "Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility", "description": "How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.", "kind": "sub", "order": 3, "html": "https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents", "markdown": "https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md", "json": "https://changegamer.ai/api/articles/serving-markdown-variants-to-ai-agents.json" }, { "slug": "how-ai-search-engines-choose-sources", "title": "How AI Search Engines Choose Sources (And What You Can Actually Influence)", "description": "What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.", "kind": "sub", "order": 4, "html": "https://changegamer.ai/articles/how-ai-search-engines-choose-sources", "markdown": "https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md", "json": "https://changegamer.ai/api/articles/how-ai-search-engines-choose-sources.json" }, { "slug": "should-you-block-ai-crawlers", "title": "Should You Block AI Crawlers? A Decision Framework by Business Model", "description": "Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.", "kind": "sub", "order": 5, "html": "https://changegamer.ai/articles/should-you-block-ai-crawlers", "markdown": "https://changegamer.ai/articles/should-you-block-ai-crawlers.md", "json": "https://changegamer.ai/api/articles/should-you-block-ai-crawlers.json" }, { "slug": "what-to-charge-ai-crawlers", "title": "What to Charge AI Crawlers: Pricing Models for Machine Buyers", "description": "Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.", "kind": "sub", "order": 6, "html": "https://changegamer.ai/articles/what-to-charge-ai-crawlers", "markdown": "https://changegamer.ai/articles/what-to-charge-ai-crawlers.md", "json": "https://changegamer.ai/api/articles/what-to-charge-ai-crawlers.json" }, { "slug": "http-402-paywall-implementation", "title": "Implementing an HTTP 402 Paywall an Agent Can Actually Pay", "description": "A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.", "kind": "sub", "order": 7, "html": "https://changegamer.ai/articles/http-402-paywall-implementation", "markdown": "https://changegamer.ai/articles/http-402-paywall-implementation.md", "json": "https://changegamer.ai/api/articles/http-402-paywall-implementation.json" }, { "slug": "structured-data-for-ai-agents", "title": "Structured Data for AI Agents: Which Schema.org Types Earn Their Keep", "description": "Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.", "kind": "sub", "order": 8, "html": "https://changegamer.ai/articles/structured-data-for-ai-agents", "markdown": "https://changegamer.ai/articles/structured-data-for-ai-agents.md", "json": "https://changegamer.ai/api/articles/structured-data-for-ai-agents.json" }, { "slug": "json-api-design-for-agents", "title": "JSON API Design for AI Agents: Endpoints They Prefer Over Scraping", "description": "How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.", "kind": "sub", "order": 9, "html": "https://changegamer.ai/articles/json-api-design-for-agents", "markdown": "https://changegamer.ai/articles/json-api-design-for-agents.md", "json": "https://changegamer.ai/api/articles/json-api-design-for-agents.json" }, { "slug": "measuring-ai-agent-traffic", "title": "Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions", "description": "Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.", "kind": "sub", "order": 10, "html": "https://changegamer.ai/articles/measuring-ai-agent-traffic", "markdown": "https://changegamer.ai/articles/measuring-ai-agent-traffic.md", "json": "https://changegamer.ai/api/articles/measuring-ai-agent-traffic.json" }, { "slug": "licensing-content-for-ai-training", "title": "Licensing Content for AI Training: RSL, Terms, and Provenance", "description": "How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.", "kind": "sub", "order": 11, "html": "https://changegamer.ai/articles/licensing-content-for-ai-training", "markdown": "https://changegamer.ai/articles/licensing-content-for-ai-training.md", "json": "https://changegamer.ai/api/articles/licensing-content-for-ai-training.json" }, { "slug": "mcp-server-as-distribution-channel", "title": "Running an MCP Server as a Distribution Channel for Your Content", "description": "Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.", "kind": "sub", "order": 12, "html": "https://changegamer.ai/articles/mcp-server-as-distribution-channel", "markdown": "https://changegamer.ai/articles/mcp-server-as-distribution-channel.md", "json": "https://changegamer.ai/api/articles/mcp-server-as-distribution-channel.json" }, { "slug": "why-ai-agents-cant-read-your-site", "title": "Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them", "description": "A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.", "kind": "sub", "order": 13, "html": "https://changegamer.ai/articles/why-ai-agents-cant-read-your-site", "markdown": "https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md", "json": "https://changegamer.ai/api/articles/why-ai-agents-cant-read-your-site.json" } ] }, "navigation": { "pillar": { "slug": "agent-ready-website", "title": "The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization", "description": "The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.", "kind": "pillar", "order": 0, "html": "https://changegamer.ai/articles/agent-ready-website", "markdown": "https://changegamer.ai/articles/agent-ready-website.md", "json": "https://changegamer.ai/api/articles/agent-ready-website.json" }, "previous": { "slug": "how-ai-search-engines-choose-sources", "title": "How AI Search Engines Choose Sources (And What You Can Actually Influence)", "description": "What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.", "kind": "sub", "order": 4, "html": "https://changegamer.ai/articles/how-ai-search-engines-choose-sources", "markdown": "https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md", "json": "https://changegamer.ai/api/articles/how-ai-search-engines-choose-sources.json" }, "next": { "slug": "what-to-charge-ai-crawlers", "title": "What to Charge AI Crawlers: Pricing Models for Machine Buyers", "description": "Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.", "kind": "sub", "order": 6, "html": "https://changegamer.ai/articles/what-to-charge-ai-crawlers", "markdown": "https://changegamer.ai/articles/what-to-charge-ai-crawlers.md", "json": "https://changegamer.ai/api/articles/what-to-charge-ai-crawlers.json" } }, "resources": [ { "slug": "ai-crawler-policy", "html": "https://changegamer.ai/resources/ai-crawler-policy", "markdown": "https://changegamer.ai/resources/ai-crawler-policy.md", "json": "https://changegamer.ai/api/resources/ai-crawler-policy.json" }, { "slug": "content-signals-explained", "html": "https://changegamer.ai/resources/content-signals-explained", "markdown": "https://changegamer.ai/resources/content-signals-explained.md", "json": "https://changegamer.ai/api/resources/content-signals-explained.json" }, { "slug": "agentic-payment-protocols", "html": "https://changegamer.ai/resources/agentic-payment-protocols", "markdown": "https://changegamer.ai/resources/agentic-payment-protocols.md", "json": "https://changegamer.ai/api/resources/agentic-payment-protocols.json" }, { "slug": "access-and-pricing", "html": "https://changegamer.ai/resources/access-and-pricing", "markdown": "https://changegamer.ai/resources/access-and-pricing.md", "json": "https://changegamer.ai/api/resources/access-and-pricing.json" } ] }