How long should an llms.txt file be?

Short enough that a model can read the whole thing before deciding what to fetch — for most sites that is 20 to 100 links. If your file is hundreds of kilobytes you have built a sitemap in Markdown, which is a different file with a different job.

Should the links point to .md files or HTML pages?

Markdown variants where they exist, HTML otherwise. The convention explicitly favours clean Markdown link targets because the point of the file is cheap consumption. If you cannot serve Markdown yet, link the HTML — an accurate index of HTML pages still beats no index.

Do I need llms-full.txt as well?

Only if your corpus is small enough to inline sensibly and you expect agents to want all of it. It is widely adopted practice rather than part of the original proposal. If you publish one, keep it generated — never hand-assembled — and be deliberate about whether paid content appears in it.

Will an llms.txt file improve my rankings?

There is no evidence that it does, and it is not a ranking factor in any published sense. It helps agents and tools that look for it. Publish it because it is an hour of work with no downside, not because you expect traffic from it.

← All guides · The agent-ready web

How to Write an llms.txt File (Format, Template, and Maintenance)

Part 2 of The agent-ready web · 1,301 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.

In short

The format is four elements in order: an H1 name, an optional blockquote summary, optional prose, then H2 sections of annotated links. Nothing else is defined.
Curate ruthlessly. A model choosing what to fetch benefits from forty good links, not four thousand — a dumped sitemap is the most common way to make the file worthless.
Point every link at a Markdown variant where you have one. The file exists to reduce token cost; linking to JavaScript-heavy HTML defeats the purpose.
Generate it from the same source that generates your pages. A hand-maintained llms.txt drifts within weeks and then lies about your site.
Put changelogs, archives and secondary material under ## Optional — agents under context pressure are permitted to skip that whole section.

Part of the The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization guide.

An llms.txt file takes about an hour to write and most of them are wrong in the same two ways: they are dumped sitemaps, or they are hand-maintained and out of date. This is the process that avoids both.

For where this file sits in the wider picture, see the pillar: the agent-ready website. For how it differs from the other root-level files, see llms.txt vs robots.txt vs sitemap.xml.

The format, exactly

The convention — proposed by Jeremy Howard (Answer.AI) on 3 September 2024 — defines four elements, in this order. Only the first is required.

H1 — the project or site name, nothing else: # Example Corp
Blockquote — one paragraph of context: > What this site is.
Free-form prose — paragraphs or bullets, no sub-headings.
H2 sections — repeatable named link lists.

Inside a section, every line is a link with an optional prose note:

- [Display name](https://example.com/page.md): what this page is for

One section name is special. ## Optional marks links agents may skip entirely when they are short on context. Everything else is treated as core.

Serve the file at exactly /llms.txt with Content-Type: text/plain; charset=utf-8. Full details, including the agent-side consumption order, are in the llms.txt convention explained.

Step 1 — Decide what the file is for

Write the answer to this in one sentence before writing any links: what task is an agent trying to complete when it lands on my site? The honest answer changes the file completely.

Documentation site: the agent is trying to use your API correctly. Lead with quickstart, auth, and the reference — not with your blog.
Publisher: the agent is trying to answer a question with a citable source. Lead with your evergreen explainers and your topic hubs, not with today's news (as of July 2026).
SaaS product: the agent is trying to decide whether you fit a requirement. Lead with what the product does, pricing, and integration limits.
Reference corpus: the agent wants a specific fact. Lead with the index and the machine formats.

Step 2 — Choose sections that map to intent

Sections are the only structure a model gets, so name them for jobs, not for your internal org chart. Patterns that work:

## Start here          → orientation, 1–3 links
## Guides              → task-shaped how-tos
## Reference           → stable, lookup-shaped material
## API                 → machine entry points (JSON, OpenAPI, MCP)
## Policy              → terms, licensing, access, pricing
## Optional            → changelog, archive, secondary material

Resist the urge to mirror your navigation. Navigation is optimised for browsing; this file is optimised for a single decision — which one URL do I fetch next?

Step 3 — Write the annotations, because they carry the file

The prose note after each link is what a model uses to choose. Two rules:

Say what the page contains, not what it is called. : installation, authentication, and a first working request beats : our quickstart guide.
Be specific enough to be exclusionary. A good note lets the model rule the page out as fast as it rules it in. : pricing tiers, per-seat limits, and enterprise terms tells a model that a question about rate limits is answered elsewhere.

Bad and good, same page:

- [Docs](https://example.com/docs): our documentation
- [API reference](https://example.com/docs/api.md): every endpoint with request and
  response shapes, error codes, and rate-limit headers

Step 4 — Point at the cheapest representation you have

Link Markdown variants where they exist. If your pages are heavy HTML with navigation, cookie banners and script tags, every link in your llms.txt is an expensive fetch and the file has failed at its one job. Shipping Markdown twins is a small change with an outsized effect — see serving Markdown variants to AI agents.

Where a JSON endpoint answers the question better than a page, link the endpoint. An llms.txt section listing /api/* entry points, each with a one-line note about what it returns, saves an agent from scraping HTML to reconstruct data you already publish structurally — see JSON API design for agents.

Step 5 — Generate it, never hand-maintain it

This is the difference between a file that stays true and a file that lies. Derive llms.txt from whatever already knows your content — CMS query, content collection, or a data module — at build time, so a new or renamed page updates the index automatically.

A minimal generator in a static build:

// pages/llms.txt.js — one source of truth, two rendered surfaces
export function GET() {
  const lines = [
    `# ${SITE.name}`, "",
    `> ${SITE.description}`, "",
  ];
  for (const section of SECTION_ORDER) {
    const pages = content.filter((p) => p.section === section && !p.draft);
    if (!pages.length) continue;
    lines.push(`## ${section}`, "");
    for (const p of pages) {
      lines.push(`- [${p.title}](${SITE.url}/${p.slug}.md): ${p.summary}`);
    }
    lines.push("");
  }
  return new Response(lines.join("\n"),
    { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}

If your stack cannot generate it, the fallback is a scheduled job that regenerates the file and opens a pull request when it changes — anything that removes the human from the loop.

Step 6 — Validate

There is no official validator, so check the things that actually break:

# 1. Served as plain text at the exact path
curl -sI https://example.com/llms.txt | head -3

# 2. Exactly one H1, at the top
curl -s https://example.com/llms.txt | grep -c "^# "

# 3. Every link resolves (and none redirects into a chain)
curl -s https://example.com/llms.txt \
  | grep -o "https://[^)]*" | sort -u \
  | while read -r u; do printf "%s %s\n" "$(curl -so /dev/null -w "%{http_code}" "$u")" "$u"; done \
  | grep -v "^200"

# 4. A crawler UA gets the same file a browser does
curl -s -A "GPTBot/1.2" https://example.com/llms.txt | head -5

Add step 3 to CI. Dead links in your machine index are worse than dead links in your footer, because an agent will not go looking for the working URL.

A copy-paste template

# Acme Analytics

> Product analytics API and SDKs. Event ingestion, funnels, retention,
> and a query API. Self-serve from €0; SOC 2 Type II.

Prefer the .md variants linked below — they are the same content without
navigation or scripts. Structured data is at /api/openapi.json.

## Start here

- [What Acme does](https://acme.com/product.md): capabilities, limits, and what it is not
- [Quickstart](https://acme.com/docs/quickstart.md): install, authenticate, send a first event

## Guides

- [Event schema design](https://acme.com/docs/events.md): naming, properties, versioning rules
- [Backfills](https://acme.com/docs/backfill.md): importing historical events, dedup semantics

## Reference

- [Query API](https://acme.com/docs/query.md): endpoints, filters, pagination, rate limits
- [SDK matrix](https://acme.com/docs/sdks.md): languages, versions, supported features

## API

- [openapi.json](https://acme.com/api/openapi.json): OpenAPI 3.1 description of every endpoint
- [status.json](https://acme.com/api/status.json): current availability and incident history

## Policy

- [Pricing](https://acme.com/pricing.md): plans, usage limits, overage rates
- [Terms and licensing](https://acme.com/terms.md): permitted use, AI training terms

## Optional

- [Changelog](https://acme.com/changelog.md): releases, newest first
- [Blog archive](https://acme.com/blog.md): posts index

Maintenance: the three things that rot

Descriptions. Pages change more often than link text. If the file is generated from the same summaries your pages use, this fixes itself.
Section membership. Material that becomes secondary should move to ## Optional, not stay in ## Guides forever.
Link targets. Renaming a URL without a redirect breaks your machine index silently. Treat llms.txt as a public API surface: stable, tested, versioned with your site.

One more discipline, borrowed from how this site publishes: never gate the index. If a paid tier exists, the index may say a resource is premium and how to buy it, but the index itself and your terms page must stay free — otherwise an agent that hits your paywall cannot discover how to pay. The onboarding contract is written out in getting started for agents.

Frequently asked questions

How long should an llms.txt file be?: Short enough that a model can read the whole thing before deciding what to fetch — for most sites that is 20 to 100 links. If your file is hundreds of kilobytes you have built a sitemap in Markdown, which is a different file with a different job.
Should the links point to .md files or HTML pages?: Markdown variants where they exist, HTML otherwise. The convention explicitly favours clean Markdown link targets because the point of the file is cheap consumption. If you cannot serve Markdown yet, link the HTML — an accurate index of HTML pages still beats no index.
Do I need llms-full.txt as well?: Only if your corpus is small enough to inline sensibly and you expect agents to want all of it. It is widely adopted practice rather than part of the original proposal. If you publish one, keep it generated — never hand-assembled — and be deliberate about whether paid content appears in it.
Will an llms.txt file improve my rankings?: There is no evidence that it does, and it is not a ranking factor in any published sense. It helps agents and tools that look for it. Publish it because it is an hour of work with no downside, not because you expect traffic from it.

#llms.txt #ai-visibility #agents #documentation

Agents: this guide is available as Markdown and JSON; the whole cluster is indexed at /api/articles.json. The reference corpus behind it is at /llms.txt, with licensing at pricing.