{ "slug": "content-signals-explained", "title": "Content Signals Explained: The robots.txt Extension for AI Usage Intent", "description": "What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL.", "category": "Reference", "tags": [ "content-signals", "robots.txt", "aipref", "rsl", "cloudflare", "standard", "agents" ], "updated": "2026-07-10", "canonical": "https://changegamer.ai/resources/content-signals-explained", "markdown": "https://changegamer.ai/resources/content-signals-explained.md", "outline": [ { "depth": 2, "text": "Key facts", "anchor": "key-facts" }, { "depth": 2, "text": "How this differs from what's already covered", "anchor": "how-this-differs-from-what-s-already-covered" }, { "depth": 2, "text": "Decision rule", "anchor": "decision-rule" }, { "depth": 2, "text": "Verified sources", "anchor": "verified-sources" } ], "related": [ { "slug": "agent-skills-explained", "title": "Agent Skills Explained: The SKILL.md Open Standard", "description": "What Agent Skills are, the exact SKILL.md field constraints, the three-level progressive-disclosure loading model, and how Skills differ from MCP tools and native function calling.", "url": "https://changegamer.ai/resources/agent-skills-explained" }, { "slug": "agentic-payment-protocols", "title": "Agentic Payment Protocols: 402, Pay Per Crawl, and x402", "description": "Implementor's comparison of the three live mechanisms for agent-to-server content payment: self-hosted HTTP 402 gates, Cloudflare Pay Per Crawl, and the x402 open standard — plus how RSL fits as the licensing layer, not the settlement layer.", "url": "https://changegamer.ai/resources/agentic-payment-protocols" }, { "slug": "agents-md-explained", "title": "AGENTS.md Explained: The Open Standard for Repo-Level Agent Instructions", "description": "What AGENTS.md is, why OpenAI created it, how it differs from SKILL.md and a human-facing README, which coding agents read it today, and what this session could and could not independently confirm about its move to the Linux Foundation.", "url": "https://changegamer.ai/resources/agents-md-explained" }, { "slug": "ai-crawler-policy", "title": "AI Crawler Policy: robots.txt and User-Agents", "description": "Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.", "url": "https://changegamer.ai/resources/ai-crawler-policy" } ], "body": "Content Signals declare *how* content a crawler already fetched may be used — not *whether* it may be fetched at all. A permission layer above the access-control tokens in /resources/ai-crawler-policy, and a lighter, non-binding cousin of the RSL license file in /resources/agentic-payment-protocols.\n\n## Key facts\n\n- Cloudflare announced the Content Signals Policy on September 24, 2025: a `Content-Signal` directive inside a robots.txt `User-agent` block, written as a comment so unaware parsers ignore it safely.\n- Three signals, each `yes`/`no`/unset: `search` (search-index use — hyperlinks and short excerpts; excludes AI-generated summaries), `ai-input` (real-time generative use, e.g. RAG/grounding), `ai-train` (training or fine-tuning a model).\n- Example: `User-agent: *` then `Content-Signal: search=yes, ai-train=no` — allow indexing, disallow training, no stated preference on `ai-input`.\n- Cloudflare's own notice says these are *preferences*, not a technical block, but that they \"might have legal rights in various jurisdictions\" — citing the EU's text-and-data-mining opt-out (Directive 2019/790).\n- Cloudflare defaulted the policy on for 3.8M+ domains on its managed robots.txt feature, to `search=yes, ai-train=no`.\n- The IETF chartered an AI Preferences (AIPREF) Working Group in 2025 to standardize the same idea as a `Content-Usage` signal (robots.txt rule or HTTP header); the individual draft `draft-romm-aipref-contentsignals` maps Cloudflare's vocabulary into that track — as of mid-2026 still in-progress, not an RFC.\n\n## How this differs from what's already covered\n\nai-crawler-policy's UA tokens and `Disallow` answer *can this bot fetch the page*. Content Signals assume the fetch happened and declare *what the publisher tolerates being done with it afterward* — usage intent, not access control. RSL goes further: a separate XML file declaring binding license terms and a price. Content Signals are the cheapest layer to add (one comment) and, like `Disallow`, voluntary-compliance-only.\n\n## Decision rule\n\n- Keep search visibility, block training: `search=yes, ai-train=no`.\n- Also opt out of live generative use: add `ai-input=no`.\n- Need an enforceable, priced license, not a preference: pair with RSL and/or HTTP 402 (/resources/paying-for-access-402) — neither replaces the other.\n\n## Verified sources\n\n- Cloudflare, \"Giving users choice with Cloudflare's new Content Signals Policy\" (fetched directly): https://blog.cloudflare.com/content-signals-policy/\n- IETF AI Preferences (AIPREF) Working Group — WebSearch-corroborated (2+ agreeing sources: The Register and APNIC coverage, plus the IETF's own blog), not fetched this session (datatracker.ietf.org returned HTTP 403): https://datatracker.ietf.org/wg/aipref/about/\n- `draft-romm-aipref-contentsignals-00`, \"Vocabulary For Expressing Content Signals\" — WebSearch-corroborated only (2 independent search results), not fetched: https://www.ietf.org/archive/id/draft-romm-aipref-contentsignals-00.html\n- See also: /resources/ai-crawler-policy, /resources/agentic-payment-protocols, /resources/web-bot-auth", "sources": [ "https://blog.cloudflare.com/content-signals-policy/", "https://datatracker.ietf.org/wg/aipref/about/", "https://www.ietf.org/archive/id/draft-romm-aipref-contentsignals-00.html" ] }