Should You Block AI Crawlers? A Decision Framework by Business Model
Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- There is no single "AI crawler" decision. There are four: training collection, search indexing, live user-triggered fetches, and open web archives. They pay you back completely differently.
- Training opt-out tokens (
Google-Extended,Applebot-Extended) cost you nothing in search visibility — they are policy tokens, not crawlers. - Blocking search-indexing bots (
OAI-SearchBot,Claude-SearchBot,PerplexityBot) removes you from those products' answers. That is a distribution decision, and usually the expensive one to get wrong. - Blocking is enforcement-free:
robots.txtis advisory, and some fetchers ignore it by design. If you need enforcement, that is a WAF rule or a 402, not aDisallow. - The middle path most publishers actually want: allow answer-engine indexing, decline training, declare usage preferences, and price bulk access.
"Should we block AI crawlers?" is the wrong question, and asking it that way is why so many sites end up with a robots.txt that costs them distribution without protecting anything. There are four decisions hiding inside it, and they have different answers for almost every business.
This is the control chapter of the agent-ready website. The maintained token reference behind it is AI crawler policy.
Step 1 — Split "AI crawler" into four classes
| Class | Example tokens | What it does for you | What it takes |
|---|---|---|---|
| Training collection | GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider |
Nothing directly — no traffic, no citation | Your content becomes model weights |
| Search / answer indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, Googlebot |
Presence in answers, some referral clicks | A fetch; sometimes answers replace the click |
| Live user fetch | ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher, MistralAI-User |
A real person asked for your page, right now | Almost nothing — this is closest to a human visit |
| Open archive | CCBot (Common Crawl) |
Indirect reach; research and downstream use | Your content feeds many models at once |
Add a fifth row that is not a crawler at all: training opt-out tokens — Google-Extended and Applebot-Extended. The fetch is still done by Googlebot and Applebot; the token controls downstream training use only. This is the cheapest decision on the page: disallowing them costs no search visibility.
Step 2 — Answer four questions, not one
Q1. Do you want to be in AI answers? For nearly every business whose customers ask questions before buying, yes. Answer engines are a discovery surface; being absent means a competitor is cited instead. Blocking search-indexing bots is the decision most likely to be regretted.
Q2. Do you want your content used as training data? This is where content-as-product businesses diverge from everyone else. If your archive is your asset, declining training use is coherent. If your content is marketing for something else you sell, training exposure is closer to free brand distribution than to theft.
Q3. Do you want live, user-triggered fetches? Almost always yes. A person asked for your page. Blocking this class is closest to blocking a browser.
Q4. Do you want bulk archival collection? Common Crawl is one fetch that feeds many downstream models and much academic work. If you decline training, blocking CCBot is the consistent follow-through; if you have not decided, this is a low-stakes decision.
Step 3 — Read the answer off your business model
Content is the product (news, research, paid archives, data)
Allow answer indexing and live fetches, decline training, and price bulk access. Blanket blocking removes you from discovery while doing nothing about content already collected, and it forgoes the only revenue path that actually exists.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Content-Signal: search=yes, ai-train=no
Sitemap: https://example.com/sitemap.xml
License: https://example.com/license.xml
Then put a price on the bulk path so the answer to "can I license this?" is machine-readable rather than a contact form: what to charge AI crawlers and licensing content for AI training.
Content is marketing (SaaS, agencies, tools, most B2B)
Allow essentially everything. Your docs being inside a model that developers ask for help is a distribution win, not a leak. The realistic risk is not theft, it is being absent from the answer when someone asks which tool solves their problem.
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes
Sitemap: https://example.com/sitemap.xml
Content is a community asset (forums, UGC, wikis)
The decision is not really yours alone — contributors have expectations, and your terms may constrain what you can license. Declining training while allowing answer indexing is the defensible default, and the reasoning belongs in public.
Content is regulated, confidential or personal
Do not rely on robots.txt at all. Advisory files are the wrong tool for material that must not be ingested: use authentication, and note that a training crawler with valid credentials is not a scenario robots.txt addresses. Data-handling obligations do not disappear because the client is a model.
Step 4 — Understand what you have and have not achieved
Three limits, all of which matter more than the syntax:
- It is advisory. Compliance is voluntary. Perplexity states
Perplexity-Userignores robots.txt by design; since a December 2025 documentation update OpenAI no longer listsChatGPT-Useramong its robots.txt-compliant agents; some crawlers have documented histories of ignoringDisallow. - It is forward-looking only. A
Disallowadded today (as of July 2026) does not retract content collected last year, nor content mirrored on sites you do not control. - User agents are spoofable. Anyone can claim to be
GPTBot. Vendor IP-range verification is the only way to know, and every major vendor publishes ranges — see the source list in AI crawler policy.
If any of those limits is unacceptable for your content, the honest options are enforcement (WAF/IP rules) or pricing (HTTP 402) — see implementing an HTTP 402 paywall and agentic payment protocols.
Step 5 — Express usage preferences, not just access
Blocking is a blunt instrument: it cannot say "index me but do not train on me". Content Signals can. Cloudflare announced the Content Signals Policy on 24 September 2025 with three directives — search, ai-input (real-time generative use such as grounding) and ai-train — and defaulted it on for over 3.8 million domains at search=yes, ai-train=no. Cloudflare is explicit that these are preferences rather than a technical block, while noting they "might have legal rights in various jurisdictions", citing the EU text-and-data-mining opt-out in Directive 2019/790. The IETF's AIPREF Working Group, chartered in 2025, is standardising the same idea; as of mid-2026 it is still in progress, not an RFC. Details: content signals explained.
Step 6 — Verify, because policy changes break access
Every robots.txt edit deserves the same follow-up, since the layer that actually decides is your edge:
for ua in "GPTBot/1.2" "OAI-SearchBot/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
printf "%-22s " "$ua"
curl -sI -A "$ua" https://example.com/ | head -1
done
A 403 where you intended an Allow means bot management is overriding your published policy — the exact failure this site hit when Cloudflare's managed AI-bots rule silently 403'd six crawlers that robots.txt explicitly allowed. Everything that can go wrong here is catalogued in why AI agents can't read your site.
The framework in one paragraph
Allow live user fetches always. Allow answer indexing unless you have a specific reason not to be discoverable. Decide training separately, and express that decision with both a Disallow and a Content Signal, because they say different things. Publish a licence if your content has licensing value, and a price if you want revenue rather than a preference. Then verify at the edge — because a policy your CDN overrides is not a policy, it is a document.
Frequently asked questions
- Will blocking GPTBot hurt my Google rankings?
- No. `GPTBot` is OpenAI's training crawler and has no relationship to Google Search. The token that matters for Google is `Googlebot` (search) versus `Google-Extended` (a training opt-out that does not affect search inclusion or ranking).
- If I block training crawlers, is my content still used for training?
- Possibly. `robots.txt` is advisory; compliance varies, some crawlers have documented histories of ignoring it, and content already collected before you blocked is not withdrawn. Blocking is a forward-looking preference, not a retraction. Content that circulates through third-party mirrors and aggregators is outside your robots.txt entirely.
- What is the difference between blocking a crawler and declaring Content Signals?
- Blocking says "do not fetch this". Content Signals say "you may fetch it, but here is what you may use it for" — `search`, `ai-input`, `ai-train`, each yes or no. Both are voluntary-compliance mechanisms; signals let you stay visible in answers while declining training use, which blocking cannot express.
- Is there a way to actually enforce this?
- Yes, at two layers: WAF/IP rules stop the request, and HTTP 402 prices it. Everything in robots.txt is a request for cooperation. Vendor IP-range verification matters here too, because user agents are trivially spoofed.