# llms.txt vs robots.txt vs sitemap.xml: Which File Does What

> The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.

Guide: The agent-ready web — part 1
Published: 2026-07-26 · Updated: 2026-07-26 · 1226 words
Canonical: https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap
JSON: https://changegamer.ai/api/articles/llms-txt-vs-robots-txt-vs-sitemap.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- `robots.txt` is **permission**, `sitemap.xml` is **inventory**, `llms.txt` is **curation**. Different questions, different consumers, no overlap in function.
- Only `robots.txt` changes crawler behaviour on compliant clients. A sitemap is a hint; llms.txt is a convenience that no vendor has committed to reading.
- You need all three for different reasons, and none of them fixes an edge rule that returns 403.
- The one file that can actively cost you traffic if written carelessly is `robots.txt` — a single misplaced `Disallow` removes you from products you wanted to be in.

---

Three files sit at the root of an agent-ready site, and they get conflated constantly — usually by someone asking whether llms.txt "replaces" the sitemap. It does not. They answer different questions for different consumers, and each one is useless at the others' job.

This is a companion to [the agent-ready website](/articles/agent-ready-website), which covers the whole stack; this page is only about the three root files.

## The one-line version

| File | Question it answers | Primary consumer | Format | Enforceable? |
|---|---|---|---|---|
| `/robots.txt` | *May you fetch this?* | Crawlers (search and AI) | Plain text directives | Advisory — honoured voluntarily |
| `/sitemap.xml` | *What URLs exist, and when did they change?* | Search-engine crawlers | XML (Sitemaps protocol) | No — a hint for discovery |
| `/llms.txt` | *What should a model read first, and why?* | LLMs and agents | Markdown | No — a convenience index |

## robots.txt — permission, per client

The oldest of the three and the only one that changes crawler behaviour. A compliant crawler fetches `/robots.txt` before anything else and honours `Disallow` rules for its own user-agent token.

What makes it a business document rather than a technical one is that AI vendors ship several tokens with different jobs. OpenAI's `GPTBot` collects training data while `OAI-SearchBot` indexes for ChatGPT Search; Anthropic splits `ClaudeBot`, `Claude-SearchBot` and `Claude-User` the same way. `Google-Extended` and `Applebot-Extended` are not crawlers at all — they are training opt-out tokens, with the actual fetching still done by `Googlebot` and `Applebot`, which is why disallowing them costs no search visibility.

```
# Search indexing: yes. Training: no.
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml
License: https://example.com/license.xml
```

Two caveats that matter more than the syntax:

1. **It is advisory.** Live user-triggered fetchers are the sharpest edge: Perplexity states `Perplexity-User` ignores robots.txt by design, and since a December 2025 documentation update OpenAI no longer lists `ChatGPT-User` among its robots.txt-compliant agents. Some crawlers have documented histories of ignoring `Disallow` outright.
2. **Your WAF is above it.** Edge rules are evaluated before the crawler can even read the file. If bot management blocks a UA, an `Allow` line changes nothing — see the failure catalogue in [why AI agents can't read your site](/articles/why-ai-agents-cant-read-your-site).

The complete, maintained token table with vendor, purpose and compliance status is in [AI crawler policy](/resources/ai-crawler-policy). Which tokens to allow is a business decision, worked through in [should you block AI crawlers?](/articles/should-you-block-ai-crawlers).

### robots.txt now carries two extra statements

Because it is the one file every crawler already fetches, robots.txt has become the mounting point for newer signals:

- **Content Signals** — `search`, `ai-input`, `ai-train`, written inside a user-agent block, declaring what fetched content may be used for rather than whether it may be fetched. Announced by Cloudflare on 24 September 2025 and defaulted on for over 3.8 million domains at `search=yes, ai-train=no`. Reference: [content signals explained](/resources/content-signals-explained).
- **`License:`** — a pointer to an RSL licence document, typically `/license.xml`, stating permitted uses and compensation terms. See [licensing content for AI training](/articles/licensing-content-for-ai-training).

## sitemap.xml — inventory and change signal

A sitemap is a machine-generated list of every canonical URL you want discovered, with optional `lastmod`, `changefreq` and `priority`. It exists so a crawler does not have to find your pages by following links, and so it can skip re-fetching pages that have not changed.

Three rules cover almost all sitemap value:

- **Complete but canonical only.** One entry per canonical URL. No redirects, no `noindex` pages, no parameter variants. A sitemap full of URLs that 301 or ask not to be indexed teaches crawlers to distrust it.
- **Real `lastmod`.** Per-URL, reflecting actual content change — not the build timestamp. A build-stamped sitemap that claims 50,000 pages changed today is worse than no `lastmod` at all (as of July 2026), because it destroys the only mechanism you have for saying "this one, actually."
- **Split at scale.** 50,000 URLs or 50 MB uncompressed per file, with a sitemap index above them.

For agents, the sitemap matters less than for search crawlers: an agent working a task does not enumerate your site, it wants the two pages that answer its question. That is exactly the gap llms.txt tries to fill.

## llms.txt — curation for a token budget

`/llms.txt` is a plain-Markdown, hand-curated index written for models: an H1 with the site name, an optional one-paragraph blockquote summary, and H2 sections of annotated links. It was proposed by Jeremy Howard (Answer.AI) on 3 September 2024 and is an emerging community convention — not an RFC, not a ratified standard, and not something any major vendor has committed to consuming.

Its value is precisely the opposite of a sitemap's: a sitemap says *everything exists*, llms.txt says *these forty things matter, and here is what each one is for*. That is the useful signal for a client with a context window.

```
# Example Corp

> API documentation and integration guides for the Example platform.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): install, authenticate, first request
- [API reference](https://example.com/docs/api.md): every endpoint with request/response shapes

## Optional

- [Changelog](https://example.com/changelog.md): release history
```

The `## Optional` section is a genuine part of the convention: agents under context pressure may skip everything in it. That makes it the right home for changelogs, archives and secondary material — and it means where you put a link is itself a signal.

A companion `/llms-full.txt` inlines the full text of your content into one file for agents that prefer a single large fetch over many small ones. The two-file pattern is widely adopted practice built on top of the original proposal, which specifies only `/llms.txt`.

Full format details and the agent-side consumption flow are in [the llms.txt convention explained](/resources/llms-txt-explained); the writing and maintenance process is [how to write an llms.txt file](/articles/how-to-write-an-llms-txt-file).

## Do you need all three?

Yes, and the reasoning is different each time.

- **robots.txt: mandatory.** Not publishing one means every crawler applies its own default, and you have expressed no policy at all. It is also where Content Signals and your licence pointer live.
- **sitemap.xml: mandatory above roughly a hundred pages.** Below that, internal linking does the job; above it, discovery and change signalling stop being reliable without one.
- **llms.txt: cheap, optional, worth doing.** An hour of work, no downside, some upside. Do not confuse publishing it with having done the work — an llms.txt full of links to JavaScript-rendered pages behind a bot-management rule helps nobody.

One more file belongs in the mental model without belonging on your website: `AGENTS.md`, which tells coding agents how to work *inside a repository* — build commands, conventions, test invocation. Different consumer, different job, frequently confused with llms.txt because both are Markdown files aimed at models. See [AGENTS.md explained](/resources/agents-md-explained).

## A 20-minute audit

```bash
for f in robots.txt sitemap.xml llms.txt; do
  printf "%-14s " "$f"
  curl -sI "https://example.com/$f" | head -1
done

# Content type matters: llms.txt should be text/plain, not text/html
curl -sI https://example.com/llms.txt | grep -i content-type

# And confirm a crawler UA gets the same answer a browser does
curl -sI -A "GPTBot/1.2" https://example.com/robots.txt | head -1
```

Then read your own `robots.txt` line by line and ask, for each token: *do I want this specific product to use my content this specific way?* That question is the whole file.

## Frequently asked questions

### Does llms.txt replace sitemap.xml?

No. A sitemap is a complete, machine-generated inventory of every indexable URL with change metadata, consumed by search crawlers. llms.txt is a short, hand-curated Markdown index of your best content written for language models, and it deliberately omits most URLs. A site with 50,000 pages needs a sitemap; its llms.txt might list forty links.

### Do AI crawlers read llms.txt?

Some tooling and some agents do; no major AI vendor has publicly committed to reading it, and it is a community convention rather than a ratified standard. Treat it as cheap insurance and a useful index for agents that ask for it — not as a channel you can rely on for visibility.

### Where do Content Signals and RSL fit?

Both attach to robots.txt rather than replacing it. Content Signals are directives inside a robots.txt user-agent block declaring what fetched content may be used for; RSL is a separate licence document discovered through a `License:` directive in robots.txt. Access, usage and licensing are three different statements.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility](https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md): How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.
- [How AI Search Engines Choose Sources (And What You Can Actually Influence)](https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md): What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [What to Charge AI Crawlers: Pricing Models for Machine Buyers](https://changegamer.ai/articles/what-to-charge-ai-crawlers.md): Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [JSON API Design for AI Agents: Endpoints They Prefer Over Scraping](https://changegamer.ai/articles/json-api-design-for-agents.md): How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.
- [Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions](https://changegamer.ai/articles/measuring-ai-agent-traffic.md): Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.
- [Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them](https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md): A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

## Reference resources

- https://changegamer.ai/resources/llms-txt-explained.md
- https://changegamer.ai/resources/ai-crawler-policy.md
- https://changegamer.ai/resources/content-signals-explained.md
- https://changegamer.ai/resources/agents-md-explained.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt