Does llms.txt replace sitemap.xml?

No. A sitemap is a complete, machine-generated inventory of every indexable URL with change metadata, consumed by search crawlers. llms.txt is a short, hand-curated Markdown index of your best content written for language models, and it deliberately omits most URLs. A site with 50,000 pages needs a sitemap; its llms.txt might list forty links.

Do AI crawlers read llms.txt?

Some tooling and some agents do; no major AI vendor has publicly committed to reading it, and it is a community convention rather than a ratified standard. Treat it as cheap insurance and a useful index for agents that ask for it — not as a channel you can rely on for visibility.

Where do Content Signals and RSL fit?

Both attach to robots.txt rather than replacing it. Content Signals are directives inside a robots.txt user-agent block declaring what fetched content may be used for; RSL is a separate licence document discovered through a `License:` directive in robots.txt. Access, usage and licensing are three different statements.

← All guides · The agent-ready web

llms.txt vs robots.txt vs sitemap.xml: Which File Does What

Part 1 of The agent-ready web · 1,226 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.

In short

robots.txt is permission, sitemap.xml is inventory, llms.txt is curation. Different questions, different consumers, no overlap in function.
Only robots.txt changes crawler behaviour on compliant clients. A sitemap is a hint; llms.txt is a convenience that no vendor has committed to reading.
You need all three for different reasons, and none of them fixes an edge rule that returns 403.
The one file that can actively cost you traffic if written carelessly is robots.txt — a single misplaced Disallow removes you from products you wanted to be in.

Part of the The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization guide.

Three files sit at the root of an agent-ready site, and they get conflated constantly — usually by someone asking whether llms.txt "replaces" the sitemap. It does not. They answer different questions for different consumers, and each one is useless at the others' job.

This is a companion to the agent-ready website, which covers the whole stack; this page is only about the three root files.

The one-line version

File	Question it answers	Primary consumer	Format	Enforceable?
`/robots.txt`	May you fetch this?	Crawlers (search and AI)	Plain text directives	Advisory — honoured voluntarily
`/sitemap.xml`	What URLs exist, and when did they change?	Search-engine crawlers	XML (Sitemaps protocol)	No — a hint for discovery
`/llms.txt`	What should a model read first, and why?	LLMs and agents	Markdown	No — a convenience index

robots.txt — permission, per client

The oldest of the three and the only one that changes crawler behaviour. A compliant crawler fetches /robots.txt before anything else and honours Disallow rules for its own user-agent token.

What makes it a business document rather than a technical one is that AI vendors ship several tokens with different jobs. OpenAI's GPTBot collects training data while OAI-SearchBot indexes for ChatGPT Search; Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User the same way. Google-Extended and Applebot-Extended are not crawlers at all — they are training opt-out tokens, with the actual fetching still done by Googlebot and Applebot, which is why disallowing them costs no search visibility.

# Search indexing: yes. Training: no.
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml
License: https://example.com/license.xml

Two caveats that matter more than the syntax:

It is advisory. Live user-triggered fetchers are the sharpest edge: Perplexity states Perplexity-User ignores robots.txt by design, and since a December 2025 documentation update OpenAI no longer lists ChatGPT-User among its robots.txt-compliant agents. Some crawlers have documented histories of ignoring Disallow outright.
Your WAF is above it. Edge rules are evaluated before the crawler can even read the file. If bot management blocks a UA, an Allow line changes nothing — see the failure catalogue in why AI agents can't read your site.

The complete, maintained token table with vendor, purpose and compliance status is in AI crawler policy. Which tokens to allow is a business decision, worked through in should you block AI crawlers?.

robots.txt now carries two extra statements

Because it is the one file every crawler already fetches, robots.txt has become the mounting point for newer signals:

Content Signals — search, ai-input, ai-train, written inside a user-agent block, declaring what fetched content may be used for rather than whether it may be fetched. Announced by Cloudflare on 24 September 2025 and defaulted on for over 3.8 million domains at search=yes, ai-train=no. Reference: content signals explained.
License: — a pointer to an RSL licence document, typically /license.xml, stating permitted uses and compensation terms. See licensing content for AI training.

sitemap.xml — inventory and change signal

A sitemap is a machine-generated list of every canonical URL you want discovered, with optional lastmod, changefreq and priority. It exists so a crawler does not have to find your pages by following links, and so it can skip re-fetching pages that have not changed.

Three rules cover almost all sitemap value:

Complete but canonical only. One entry per canonical URL. No redirects, no noindex pages, no parameter variants. A sitemap full of URLs that 301 or ask not to be indexed teaches crawlers to distrust it.
Real lastmod. Per-URL, reflecting actual content change — not the build timestamp. A build-stamped sitemap that claims 50,000 pages changed today is worse than no lastmod at all (as of July 2026), because it destroys the only mechanism you have for saying "this one, actually."
Split at scale. 50,000 URLs or 50 MB uncompressed per file, with a sitemap index above them.

For agents, the sitemap matters less than for search crawlers: an agent working a task does not enumerate your site, it wants the two pages that answer its question. That is exactly the gap llms.txt tries to fill.

llms.txt — curation for a token budget

/llms.txt is a plain-Markdown, hand-curated index written for models: an H1 with the site name, an optional one-paragraph blockquote summary, and H2 sections of annotated links. It was proposed by Jeremy Howard (Answer.AI) on 3 September 2024 and is an emerging community convention — not an RFC, not a ratified standard, and not something any major vendor has committed to consuming.

Its value is precisely the opposite of a sitemap's: a sitemap says everything exists, llms.txt says these forty things matter, and here is what each one is for. That is the useful signal for a client with a context window.

# Example Corp

> API documentation and integration guides for the Example platform.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): install, authenticate, first request
- [API reference](https://example.com/docs/api.md): every endpoint with request/response shapes

## Optional

- [Changelog](https://example.com/changelog.md): release history

The ## Optional section is a genuine part of the convention: agents under context pressure may skip everything in it. That makes it the right home for changelogs, archives and secondary material — and it means where you put a link is itself a signal.

A companion /llms-full.txt inlines the full text of your content into one file for agents that prefer a single large fetch over many small ones. The two-file pattern is widely adopted practice built on top of the original proposal, which specifies only /llms.txt.

Full format details and the agent-side consumption flow are in the llms.txt convention explained; the writing and maintenance process is how to write an llms.txt file.

Do you need all three?

Yes, and the reasoning is different each time.

robots.txt: mandatory. Not publishing one means every crawler applies its own default, and you have expressed no policy at all. It is also where Content Signals and your licence pointer live.
sitemap.xml: mandatory above roughly a hundred pages. Below that, internal linking does the job; above it, discovery and change signalling stop being reliable without one.
llms.txt: cheap, optional, worth doing. An hour of work, no downside, some upside. Do not confuse publishing it with having done the work — an llms.txt full of links to JavaScript-rendered pages behind a bot-management rule helps nobody.

One more file belongs in the mental model without belonging on your website: AGENTS.md, which tells coding agents how to work inside a repository — build commands, conventions, test invocation. Different consumer, different job, frequently confused with llms.txt because both are Markdown files aimed at models. See AGENTS.md explained.

A 20-minute audit

for f in robots.txt sitemap.xml llms.txt; do
  printf "%-14s " "$f"
  curl -sI "https://example.com/$f" | head -1
done

# Content type matters: llms.txt should be text/plain, not text/html
curl -sI https://example.com/llms.txt | grep -i content-type

# And confirm a crawler UA gets the same answer a browser does
curl -sI -A "GPTBot/1.2" https://example.com/robots.txt | head -1

Then read your own robots.txt line by line and ask, for each token: do I want this specific product to use my content this specific way? That question is the whole file.

Frequently asked questions

Does llms.txt replace sitemap.xml?: No. A sitemap is a complete, machine-generated inventory of every indexable URL with change metadata, consumed by search crawlers. llms.txt is a short, hand-curated Markdown index of your best content written for language models, and it deliberately omits most URLs. A site with 50,000 pages needs a sitemap; its llms.txt might list forty links.
Do AI crawlers read llms.txt?: Some tooling and some agents do; no major AI vendor has publicly committed to reading it, and it is a community convention rather than a ratified standard. Treat it as cheap insurance and a useful index for agents that ask for it — not as a channel you can rely on for visibility.
Where do Content Signals and RSL fit?: Both attach to robots.txt rather than replacing it. Content Signals are directives inside a robots.txt user-agent block declaring what fetched content may be used for; RSL is a separate licence document discovered through a `License:` directive in robots.txt. Access, usage and licensing are three different statements.

#llms.txt #robots.txt #sitemap #crawlers #ai-visibility

Agents: this guide is available as Markdown and JSON; the whole cluster is indexed at /api/articles.json. The reference corpus behind it is at /llms.txt, with licensing at pricing.