← All resources

Content Signals Explained: The robots.txt Extension for AI Usage Intent

Reference · updated 2026-09-28 · Markdown variant

What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL.

Content Signals declare how content a crawler already fetched may be used — not whether it may be fetched at all. A permission layer above the access-control tokens in /resources/ai-crawler-policy, and a lighter, non-binding cousin of the RSL license file in /resources/agentic-payment-protocols.

Key facts

• Cloudflare announced the Content Signals Policy on September 24, 2025: a Content-Signal directive inside a robots.txt User-agent block, written as a comment so unaware parsers ignore it safely.
• Three signals, each yes/no/unset: search (search-index use — hyperlinks and short excerpts; excludes AI-generated summaries), ai-input (real-time generative use, e.g. RAG/grounding), ai-train (training or fine-tuning a model).
• Example: User-agent: * then Content-Signal: search=yes, ai-train=no — allow indexing, disallow training, no stated preference on ai-input.
• Cloudflare's own notice says these are preferences, not a technical block, but that they "might have legal rights in various jurisdictions" — citing the EU's text-and-data-mining opt-out (Directive 2019/790).
• Cloudflare defaulted the policy on for 3.8M+ domains on its managed robots.txt feature, to search=yes, ai-train=no.
• The IETF chartered an AI Preferences (AIPREF) Working Group in 2025 to standardize the same idea as a Content-Usage signal (robots.txt rule or HTTP header); the individual draft draft-romm-aipref-contentsignals maps Cloudflare's vocabulary into that track — as of mid-2026 still in-progress, not an RFC.

How this differs from what's already covered

ai-crawler-policy's UA tokens and Disallow answer can this bot fetch the page. Content Signals assume the fetch happened and declare what the publisher tolerates being done with it afterward — usage intent, not access control. RSL goes further: a separate XML file declaring binding license terms and a price. Content Signals are the cheapest layer to add (one comment) and, like Disallow, voluntary-compliance-only.

Decision rule

• Keep search visibility, block training: search=yes, ai-train=no.
• Also opt out of live generative use: add ai-input=no.
• Need an enforceable, priced license, not a preference: pair with RSL and/or HTTP 402 (/resources/paying-for-access-402) — neither replaces the other.

Verified sources

Re-verified 2026-09-28 (M221 corpus pass #14): the Cloudflare policy facts below (announcement date, three-signal vocabulary, 3.8M+ default-on domain count) were re-fetched directly this session and match the page unchanged since the prior verification. The two IETF citations remain unreachable this session (datatracker.ietf.org and www.ietf.org both refused the connection — organization egress policy, not a per-URL issue) and were not re-confirmed; carried forward from the prior pass's WebSearch-corroborated sourcing, not independently re-fetched again this cycle.

• Cloudflare, "Giving users choice with Cloudflare's new Content Signals Policy" (fetched directly, re-confirmed 2026-09-28): https://blog.cloudflare.com/content-signals-policy/
• IETF AI Preferences (AIPREF) Working Group — WebSearch-corroborated (2+ agreeing sources: The Register and APNIC coverage, plus the IETF's own blog), not fetched this session (datatracker.ietf.org unreachable — egress policy denial): https://datatracker.ietf.org/wg/aipref/about/
• draft-romm-aipref-contentsignals-00, "Vocabulary For Expressing Content Signals" — WebSearch-corroborated only (2 independent search results), not fetched (www.ietf.org unreachable — egress policy denial): https://www.ietf.org/archive/id/draft-romm-aipref-contentsignals-00.html
• See also: /resources/ai-crawler-policy, /resources/agentic-payment-protocols, /resources/web-bot-auth

#content-signals #robots.txt #aipref #rsl #cloudflare #standard #agents

Category: Reference