# Content Signals Explained: The robots.txt Extension for AI Usage Intent > What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL. Category: Reference · Updated: 2026-07-10 · Tags: content-signals, robots.txt, aipref, rsl, cloudflare, standard, agents Canonical: https://changegamer.ai/resources/content-signals-explained Variants: [HTML](https://changegamer.ai/resources/content-signals-explained) · [JSON](https://changegamer.ai/api/resources/content-signals-explained.json) License: https://changegamer.ai/license.xml · Access: free Content Signals declare *how* content a crawler already fetched may be used — not *whether* it may be fetched at all. A permission layer above the access-control tokens in /resources/ai-crawler-policy, and a lighter, non-binding cousin of the RSL license file in /resources/agentic-payment-protocols. ## Key facts - Cloudflare announced the Content Signals Policy on September 24, 2025: a `Content-Signal` directive inside a robots.txt `User-agent` block, written as a comment so unaware parsers ignore it safely. - Three signals, each `yes`/`no`/unset: `search` (search-index use — hyperlinks and short excerpts; excludes AI-generated summaries), `ai-input` (real-time generative use, e.g. RAG/grounding), `ai-train` (training or fine-tuning a model). - Example: `User-agent: *` then `Content-Signal: search=yes, ai-train=no` — allow indexing, disallow training, no stated preference on `ai-input`. - Cloudflare's own notice says these are *preferences*, not a technical block, but that they "might have legal rights in various jurisdictions" — citing the EU's text-and-data-mining opt-out (Directive 2019/790). - Cloudflare defaulted the policy on for 3.8M+ domains on its managed robots.txt feature, to `search=yes, ai-train=no`. - The IETF chartered an AI Preferences (AIPREF) Working Group in 2025 to standardize the same idea as a `Content-Usage` signal (robots.txt rule or HTTP header); the individual draft `draft-romm-aipref-contentsignals` maps Cloudflare's vocabulary into that track — as of mid-2026 still in-progress, not an RFC. ## How this differs from what's already covered ai-crawler-policy's UA tokens and `Disallow` answer *can this bot fetch the page*. Content Signals assume the fetch happened and declare *what the publisher tolerates being done with it afterward* — usage intent, not access control. RSL goes further: a separate XML file declaring binding license terms and a price. Content Signals are the cheapest layer to add (one comment) and, like `Disallow`, voluntary-compliance-only. ## Decision rule - Keep search visibility, block training: `search=yes, ai-train=no`. - Also opt out of live generative use: add `ai-input=no`. - Need an enforceable, priced license, not a preference: pair with RSL and/or HTTP 402 (/resources/paying-for-access-402) — neither replaces the other. ## Verified sources - Cloudflare, "Giving users choice with Cloudflare's new Content Signals Policy" (fetched directly): https://blog.cloudflare.com/content-signals-policy/ - IETF AI Preferences (AIPREF) Working Group — WebSearch-corroborated (2+ agreeing sources: The Register and APNIC coverage, plus the IETF's own blog), not fetched this session (datatracker.ietf.org returned HTTP 403): https://datatracker.ietf.org/wg/aipref/about/ - `draft-romm-aipref-contentsignals-00`, "Vocabulary For Expressing Content Signals" — WebSearch-corroborated only (2 independent search results), not fetched: https://www.ietf.org/archive/id/draft-romm-aipref-contentsignals-00.html - See also: /resources/ai-crawler-policy, /resources/agentic-payment-protocols, /resources/web-bot-auth --- ## Related resources - [Agent Skills Explained: The SKILL.md Open Standard](https://changegamer.ai/resources/agent-skills-explained.md): What Agent Skills are, the exact SKILL.md field constraints, the three-level progressive-disclosure loading model, and how Skills differ from MCP tools and native function calling. - [Agentic Payment Protocols: 402, Pay Per Crawl, and x402](https://changegamer.ai/resources/agentic-payment-protocols.md): Implementor's comparison of the three live mechanisms for agent-to-server content payment: self-hosted HTTP 402 gates, Cloudflare Pay Per Crawl, and the x402 open standard — plus how RSL fits as the licensing layer, not the settlement layer. - [AGENTS.md Explained: The Open Standard for Repo-Level Agent Instructions](https://changegamer.ai/resources/agents-md-explained.md): What AGENTS.md is, why OpenAI created it, how it differs from SKILL.md and a human-facing README, which coding agents read it today, and what this session could and could not independently confirm about its move to the Linux Foundation. - [AI Crawler Policy: robots.txt and User-Agents](https://changegamer.ai/resources/ai-crawler-policy.md): Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge. --- Index of all resources: https://changegamer.ai/llms.txt · Full corpus: https://changegamer.ai/llms-full.txt · Corpus data (NDJSON): https://changegamer.ai/api/corpus.jsonl · Offers: https://changegamer.ai/api/pricing.json