{
  "slug": "llms-txt-explained",
  "title": "The llms.txt Convention Explained",
  "description": "What llms.txt is, its exact file format, how agents consume it, and how sites should serve it.",
  "category": "Reference",
  "tags": [
    "llms.txt",
    "standard",
    "agents",
    "crawling"
  ],
  "updated": "2026-07-29",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/llms-txt-explained",
  "markdown": "https://changegamer.ai/resources/llms-txt-explained.md",
  "outline": [
    {
      "depth": 2,
      "text": "File format",
      "anchor": "file-format"
    },
    {
      "depth": 2,
      "text": "llms.txt vs llms-full.txt",
      "anchor": "llms-txt-vs-llms-full-txt"
    },
    {
      "depth": 2,
      "text": "Agent consumption guide",
      "anchor": "agent-consumption-guide"
    },
    {
      "depth": 2,
      "text": "Site emission guide",
      "anchor": "site-emission-guide"
    },
    {
      "depth": 2,
      "text": "ChangeGamer's implementation",
      "anchor": "changegamer-s-implementation"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "agent-skills-explained",
      "title": "Agent Skills Explained: The SKILL.md Open Standard",
      "description": "What Agent Skills are, the exact SKILL.md field constraints, the three-level progressive-disclosure loading model, and how Skills differ from MCP tools and native function calling.",
      "url": "https://changegamer.ai/resources/agent-skills-explained"
    },
    {
      "slug": "agents-md-explained",
      "title": "AGENTS.md Explained: The Open Standard for Repo-Level Agent Instructions",
      "description": "What AGENTS.md is, why OpenAI created it, how it differs from SKILL.md and a human-facing README, which coding agents read it today, and what this session could and could not independently confirm about its move to the Linux Foundation.",
      "url": "https://changegamer.ai/resources/agents-md-explained"
    },
    {
      "slug": "c2pa-content-credentials",
      "title": "C2PA Content Credentials: Verifying Media Provenance and AI-Generation Claims",
      "description": "How the C2PA standard cryptographically signs images, video, and audio with provenance manifests recording capture, edit, and AI-generation history — what a manifest contains, how a verifier checks one, and why a missing manifest proves nothing either way.",
      "url": "https://changegamer.ai/resources/c2pa-content-credentials"
    },
    {
      "slug": "content-signals-explained",
      "title": "Content Signals Explained: The robots.txt Extension for AI Usage Intent",
      "description": "What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL.",
      "url": "https://changegamer.ai/resources/content-signals-explained"
    }
  ],
  "furtherReading": [
    {
      "slug": "discoverability-for-paid-agent-apis",
      "title": "How Do AI Agents Discover Paid APIs? A Guide to Every Surface",
      "description": "How an AI agent finds out a paid API or resource exists before it ever reads a price: llms.txt, the JSON API index, MCP registries, incidental 402 discovery, x402 auto-listing, and what .well-known does and does not cover.",
      "url": "https://changegamer.ai/articles/discoverability-for-paid-agent-apis"
    },
    {
      "slug": "how-to-write-an-llms-txt-file",
      "title": "How to Write an llms.txt File (Format, Template, and Maintenance)",
      "description": "A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.",
      "url": "https://changegamer.ai/articles/how-to-write-an-llms-txt-file"
    }
  ],
  "body": "The `/llms.txt` convention was proposed by Jeremy Howard (Answer.AI) on September 3, 2024. It is an emerging community convention, not a ratified standard or RFC. The goal: give LLMs and autonomous agents a cheap, unambiguous entry point into a site's content — a curated index rather than a raw HTML crawl.\n\n## File format\n\nThe file lives at the site root: `https://example.com/llms.txt`. It is plain Markdown. Required and optional elements, in order:\n\n1. **H1 title** (required) — project or site name only: `# My Project`\n2. **Blockquote summary** (optional) — one-paragraph context: `> What this site is.`\n3. **Free-form prose** (optional) — paragraphs or bullet lists, no sub-headings.\n4. **H2 sections** (optional, repeatable) — each is a named link list:\n\n```\n## Docs\n\n- [Getting Started](https://example.com/start.md): installation and first run\n- [API Reference](https://example.com/api.md): full endpoint list\n\n## Optional\n\n- [Changelog](https://example.com/changelog.md)\n```\n\nThe `## Optional` section is special: agents **may skip all links in it** when operating under token or context-window pressure. Use it for secondary material (changelogs, FAQs, older guides) that is useful but not critical.\n\nLink format inside every section: `- [display name](url): optional prose note`\n\n## llms.txt vs llms-full.txt\n\n| File | Purpose |\n|------|---------|\n| `/llms.txt` | Compact index — H1, summary, and curated link lists only. Fetch first. |\n| `/llms-full.txt` | Full corpus — all page content concatenated into one file. Fetch when you need the whole site in one round trip. |\n\nThe two-file pattern is a widely-adopted practice (used by Mintlify, ChangeGamer, and others) built on top of the core proposal, which specifies only `/llms.txt`. FastHTML — the original reference implementation named in the 2024 proposal — instead expands to `llms-ctx.txt` and `llms-ctx-full.txt` via its own `llms_txt2ctx` tool, not literally an `llms-full.txt` file; treat it as the same pattern under different filenames.\n\n## Agent consumption guide\n\n1. `GET /llms.txt` — parse the H1 (site name), blockquote (summary), and link lists (available resources with descriptions).\n2. Follow individual links to fetch specific pages. Prefer `.md` URL variants when advertised — they are navigation-free and token-cheap.\n3. Under token pressure: omit all links listed under `## Optional` and skip fetching `/llms-full.txt`.\n4. `GET /llms-full.txt` when you need the full corpus in a single request (e.g., summarising the whole site or priming a long-context session).\n\n## Site emission guide\n\n- Serve `/llms.txt` as `text/plain; charset=utf-8` at the exact root path.\n- Keep it stable; treat it as a public API surface.\n- Link targets should point at clean Markdown variants of pages (not HTML), where available.\n- Optionally serve `/llms-full.txt` as concatenated Markdown for agents that prefer one large fetch.\n\n## ChangeGamer's implementation\n\nChangeGamer serves both files. `/llms.txt` lists every resource as a `.md` variant link (e.g. `https://changegamer.ai/resources/getting-started.md`). `/llms-full.txt` inlines the full body of every free resource and a summary stub for premium (paywalled) resources in a single response. Both are statically generated at build time from the same `resources.ts` source.\n\n## Verified sources\n\n- AnswerDotAI/llms-txt — canonical spec repo backing llmstxt.org: https://github.com/AnswerDotAI/llms-txt\n- llms.txt proposal source text (author, date, \"open for community input\" status, named adopters): https://raw.githubusercontent.com/AnswerDotAI/llms-txt/main/nbs/index.qmd\n- Mintlify docs source — confirms automatic hosting of both `/llms.txt` and `/llms-full.txt`: https://raw.githubusercontent.com/mintlify/docs/main/ai/llmstxt.mdx\n- FastHTML repo — confirms `/llms.txt` support and the `llms-ctx.txt` / `llms-ctx-full.txt` expansion (not `llms-full.txt`): https://github.com/AnswerDotAI/fasthtml",
  "sources": [
    "https://github.com/AnswerDotAI/llms-txt",
    "https://raw.githubusercontent.com/AnswerDotAI/llms-txt/main/nbs/index.qmd",
    "https://raw.githubusercontent.com/mintlify/docs/main/ai/llmstxt.mdx",
    "https://github.com/AnswerDotAI/fasthtml"
  ]
}