Licensing Content for AI Training: RSL, Terms, and Provenance
How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- Four layers, four different jobs:
robots.txtcontrols access, Content Signals declare usage preference, a licence states terms, and a payment gate enforces a price. None substitutes for another. - RSL (Really Simple Licensing) is a machine-readable licence document, typically an XML file discovered via a
License:directive in robots.txt. It declares terms and compensation; it does not collect money. - A licence is only useful if it is discoverable, free to read, unambiguous about the unit being licensed, and versioned.
- Provenance standards (C2PA Content Credentials) prove where an asset came from. That is a different guarantee from a licence and increasingly asked for alongside one.
- Publish the licence even if you never sell anything: it converts "we assume this is allowed" into a documented position.
Every site already has an implicit answer to "may AI companies use this?" — usually "nobody asked". Publishing a licence replaces the guess with a statement. This is the rights layer of the agent-ready website.
The four layers, ordered by force
| Layer | Where it lives | What it says | Enforced by |
|---|---|---|---|
| Access | robots.txt Disallow |
"Do not fetch this" | Voluntary compliance |
| Usage preference | Content Signals in robots.txt |
"You may fetch it; here is what you may use it for" | Voluntary compliance; possible legal relevance |
| Licence | RSL document (e.g. /license.xml) |
"These are the terms and the compensation" | Contract/copyright law, if pursued |
| Price | HTTP 402 / paid API | "Pay this to get it" | Actually enforced by your server |
Sites conflate these constantly. A Disallow is not a licence. A licence is not a paywall. Content Signals — the search / ai-input / ai-train directives Cloudflare announced on 24 September 2025 and defaulted on for over 3.8 million domains at search=yes, ai-train=no — are preferences, and Cloudflare says so explicitly while noting they "might have legal rights in various jurisdictions", citing the EU text-and-data-mining opt-out in Directive 2019/790. Details in content signals explained.
Use all four, deliberately, and make sure they agree. Contradiction between layers is the thing a counterparty will point at.
RSL: a licence a machine can read
RSL (Really Simple Licensing) is an XML licence document — commonly served at a path such as /license.xml — declaring licensing terms, usage boundaries and compensation requirements. It is discovered through a License: directive in robots.txt, and can also be advertised via HTTP headers, RSS, or an HTML <link>. The spec lives at rslstandard.org; it was announced in 2025 and remains in early adoption as of 2026.
# robots.txt
License: https://example.com/license.xml
What to make sure your document actually pins down — this is where most published licences are too vague to be usable:
- The unit. A single document? A category? The complete corpus as one deliverable? "Our content" is not a unit.
- The rights, separately. Training a model, retrieval-time grounding (RAG), indexing for search, redistribution, and creating derivative works are five different permissions. Grant or withhold each explicitly.
- The licensee. Named entity, or anyone who pays? Named licensees are what enterprise buyers expect; open terms are simpler to operate.
- Term and updates. Perpetual for the snapshot bought, or a subscription including future updates? Buyers of a corpus care enormously about this and will ask.
- Compensation. A price, a pointer to a price, or "contact us". The first two are machine-actionable; the third is where machine buyers stop.
- Attribution requirements, if any, in a form a generated answer can satisfy.
Keep the document free to fetch, always. A licence behind a paywall cannot be evaluated, which means it cannot be accepted.
Human terms and machine terms must match
Publish two representations of one policy: the machine-readable licence and a prose page a person can read and quote. Generate them from the same source if you can, and reference each in the other. When they disagree — because someone updated the prose and forgot the XML — the disagreement itself becomes the story.
Practical structure that works:
/license.xml— the RSL document, referenced fromrobots.txt./termsand/terms.md— the same terms as prose, with a Markdown twin so agents can read them cheaply./api/pricing.json— the machine-readable offer catalogue, if you sell access.- Every
402response body linking to all three.
The worked example this site publishes — free tier, paid tiers, what each licence grant actually permits, and the promise that index files and terms are never gated — is in access and pricing for crawlers.
Provenance: proving origin, not permission
A licence says what may be done with your content. Provenance says this really is your content, and here is its history. The two are complementary and increasingly requested together, particularly for images, documents and anything that could be fabricated.
C2PA Content Credentials attach cryptographically signed assertions about origin and edit history to an asset, verifiable independently of the site serving it — see C2PA content credentials. The broader question of how provenance travels through data supply chains, and where the guarantees break, is covered in AI supply chain provenance.
For text-heavy publishers the practical version is more mundane and still valuable: dated revisions, a real changelog, stable canonical URLs, and a corpus export with a version number. Those are the things that let a buyer prove which version of your content they licensed.
What licensing gets you, honestly
Three realistic outcomes, in descending order of likelihood:
- A documented position. Cheap, immediate, and useful the first time anyone asks. It also removes the "we assumed it was fine" defence.
- A usable answer to inbound interest. When a buyer with a legal department appears, the difference between a licence file plus a price and a contact form is weeks of process.
- Revenue. Real, but concentrated among sites with scarce content — proprietary data, verified freshness, structure nobody else provides. The arithmetic is in what to charge AI crawlers.
What it does not get you is enforcement. Enforcement is a WAF rule or a 402 gate, and the mechanisms for the latter are compared in agentic payment protocols with the implementation in implementing an HTTP 402 paywall.
A minimum viable rights stack
- Decide your position on training, grounding and redistribution — three separate answers.
- Express access in
robots.txtper user-agent token (should you block AI crawlers?). - Express usage preference with Content Signals in the same file.
- Publish an RSL document and point at it with
License:. - Publish prose terms saying the same thing, with a
.mdtwin. - If you sell anything, publish a machine-readable offer catalogue and link it from every 402.
- Version all of it, and keep old versions resolvable — a buyer needs to prove what the terms said on the day they agreed.
None of that requires a lawyer to start, and all of it makes the eventual lawyer conversation shorter.
Frequently asked questions
- Does a licence file stop AI companies training on my content?
- Not mechanically. Like robots.txt and Content Signals, it is a declaration rather than an enforcement mechanism. Its value is that it states terms in a form a machine can read and a human can cite, which is the precondition for any conversation — commercial or legal — about use.
- Is RSL a standard?
- It is a published specification (rslstandard.org), announced in 2025 and in early adoption as of 2026 — not an IETF or W3C standard. Publishing one is cheap and low-risk; assuming universal support for it is not.
- Do I still need terms of service language?
- Yes. The machine-readable document and your human-readable terms should say the same thing and reference each other. A licence file that contradicts your terms of service is worse than either alone.
- What should the licence actually license?
- Be explicit about the unit and the rights: individual documents versus the whole corpus, training versus retrieval-time grounding versus redistribution, named licensee versus anyone, perpetual versus term-limited, updates included or not. Vagueness here is what makes licences unusable to a buyer with a legal department.