# Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility

> How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.

Guide: The agent-ready web — part 3
Published: 2026-07-26 · Updated: 2026-07-26 · 1091 words
Canonical: https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents
JSON: https://changegamer.ai/api/articles/serving-markdown-variants-to-ai-agents.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- A Markdown twin of an HTML page removes navigation, scripts, styling and ads from what a model sees — typically a large reduction in tokens with no loss of meaning.
- Use a predictable URL suffix (`/page` → `/page.md`). It is discoverable, cacheable, linkable and works with every CDN. Content negotiation is a nice addition, not a substitute.
- Generate Markdown from your source content, not from rendered HTML. HTML-to-Markdown conversion drags navigation and boilerplate back in — the exact thing you were removing.
- Advertise it: `<link rel="alternate" type="text/markdown">` in the HTML head, the `.md` URL in your llms.txt, and a `Link` header if you can.
- Keep the two in lockstep by rendering both from one source. Two hand-maintained copies means shipping two versions of the truth.

---

Of everything in [the agent-ready website](/articles/agent-ready-website), this is the change with the best ratio of effort to effect: publish a clean Markdown version of each content page and tell clients it exists.

The reason is arithmetic. A model does not read your page, it pays for it. Every token spent on your navigation, your cookie notice, your inline SVG icons and your analytics snippet is a token not spent on your content — and when the budget runs out, the model summarises from whatever it managed to load. Markdown deletes the overhead without deleting anything that carries meaning.

## What actually gets removed

Take a typical article page and strip it to Markdown. What disappears:

- header and footer navigation, breadcrumb bars, sidebars, related-post rails
- `<script>` and `<style>` blocks, inline critical CSS, JSON-LD you already expose elsewhere
- cookie/consent markup, newsletter modals, share buttons, ad slots
- wrapper `<div>` scaffolding, utility classes, data attributes

What survives: headings, paragraphs, lists, tables, code blocks, links, emphasis. In other words, the document. The gain varies with how heavy your template is, and on a modern JS-framework page it can be most of the payload.

There is a second, subtler gain: **unambiguous structure.** In HTML a model has to infer that a bold `<div>` is a heading. In Markdown a heading is `##`. Inference is where extraction errors — and therefore misquotations of your content — come from. For the broader format comparison (Markdown vs JSON vs JSONL vs plain text, and when each is the right answer) see [data formats and schema](/resources/data-formats).

## The URL pattern

Use a suffix on the canonical path:

```
https://example.com/blog/agent-traffic          → HTML
https://example.com/blog/agent-traffic.md       → Markdown
```

Why a suffix beats the alternatives:

| Approach | Discoverable | Cacheable | Linkable from llms.txt | Notes |
|---|---|---|---|---|
| `/page.md` suffix | Yes | Yes | Yes | Recommended default |
| `Accept: text/markdown` on `/page` | Only if documented | Needs `Vary: Accept` | No | Good addition, poor primary |
| `?format=md` query | Yes | Often cache-bypassed | Yes | Query strings get stripped by tooling |
| Separate `md.example.com` host | Yes | Yes | Yes | Extra DNS/TLS/CORS surface for no gain |

Serve it as `Content-Type: text/markdown; charset=utf-8`. Do not serve Markdown as `text/plain` if you can avoid it, and never as `text/html`.

## Make it discoverable

A variant nobody can find is wasted work. Announce it in three places:

```html
<!-- 1. In the HTML head of the canonical page -->
<link rel="alternate" type="text/markdown" href="/blog/agent-traffic.md">
```

```http
# 2. As a response header, so a HEAD request is enough to learn about it
Link: </blog/agent-traffic.md>; rel="alternate"; type="text/markdown"
```

```
# 3. In llms.txt, where the link target IS the .md variant
- [Agent traffic](https://example.com/blog/agent-traffic.md): what our logs show about AI crawlers
```

The third is the one that matters most in practice, because an agent that starts at [llms.txt](/resources/llms-txt-explained) then never touches your HTML at all.

## Generate from source, not from HTML

This is the trap. The tempting implementation is a route that fetches the rendered page and runs it through an HTML-to-Markdown converter. It is fast to build and it reintroduces exactly what you were trying to remove: the converter faithfully preserves your navigation as a bullet list, your footer as a link soup, and your cookie banner as a paragraph.

Generate from the content source instead:

```js
// Astro-style: one entry, two rendered surfaces
// pages/blog/[slug].astro  → HTML (layout + nav + styles)
// pages/blog/[slug].md.ts  → Markdown (body only)
export const GET = ({ props }) => {
  const p = props.post;
  const out = [
    `# ${p.title}`, "",
    `> ${p.description}`, "",
    `Updated: ${p.updated} · Canonical: ${SITE.url}/blog/${p.slug}`, "",
    p.body,                       // the SAME markdown the HTML page renders
  ].join("\n");
  return new Response(out, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Link": `<${SITE.url}/blog/${p.slug}>; rel="canonical"`,
    },
  });
};
```

If your CMS stores rich text rather than Markdown, convert once at ingest or at build from the structured representation — not from the final DOM.

## What to include in the variant

Minimal, but not naked. A model that fetches only the `.md` should still know what it has:

1. **Title as H1.**
2. **One-line summary** (blockquote or plain line).
3. **Provenance line:** last-updated date and the canonical HTML URL. This is what lets an agent cite you correctly rather than citing "a webpage".
4. **The body**, headings intact.
5. **Optionally** a short list of related URLs at the end — internal linking works on machine readers too.

What to leave out: navigation, calls to action repeated from the template, tracking parameters on links, and anything you would not want quoted verbatim.

## Canonicalisation and indexing

Markdown has no `<head>`, so canonical signals have to travel in headers:

```http
Content-Type: text/markdown; charset=utf-8
Link: <https://example.com/blog/agent-traffic>; rel="canonical"
```

Treat the HTML page as the indexable representation, keep the pair explicitly linked with `rel="alternate"`, and include only the HTML URL in your sitemap. Some sites additionally set `X-Robots-Tag: noindex` on `.md` variants; that is defensible if you are worried about split signals, but it also tells crawlers not to index the cheapest version of your content — decide deliberately rather than by default.

## Verify it works

```bash
# Right content type, no HTML leakage
curl -sI https://example.com/blog/agent-traffic.md | grep -i content-type
curl -s  https://example.com/blog/agent-traffic.md | grep -c "<div"   # expect 0

# Same content, far smaller payload
for u in /blog/agent-traffic /blog/agent-traffic.md; do
  printf "%-28s %s bytes\n" "$u" \
    "$(curl -so /dev/null -w "%{size_download}" "https://example.com$u")"
done

# Discoverable from the HTML page
curl -s https://example.com/blog/agent-traffic | grep "text/markdown"
```

Then add two CI assertions: every HTML page has a resolvable `.md` twin, and every `.md` twin contains the page's first heading. That is enough to catch the drift that otherwise creeps in during template refactors.

## When Markdown is the wrong answer

Markdown is for documents. For everything that is really data — price lists, catalogues, availability, changelogs, search results — a JSON endpoint is better on both sides: smaller, unambiguous, and versionable. Publishing both is normal; the rule of thumb is *prose in Markdown, records in JSON*. See [JSON API design for agents](/articles/json-api-design-for-agents) and the shape this site publishes in [JSON API for agents](/resources/json-api).

And for very large corpora, one file per page is not always what an agent wants: a single concatenated `/llms-full.txt`, or NDJSON of the whole corpus, saves it hundreds of round trips. The onboarding contract that explains all of these variants to an agent in one place is [getting started for agents](/resources/getting-started).

## Frequently asked questions

### Why not just let agents parse my HTML?

They can, and they do — at a cost. A typical content page carries navigation, footers, cookie notices, inline scripts and styling that contribute nothing to meaning but consume the model's context budget and add extraction ambiguity. Markdown removes that overhead deterministically, which makes your content both cheaper and less likely to be misread.

### Should I use /page.md or content negotiation?

Ship the URL suffix first. A distinct URL is discoverable, linkable from llms.txt, cacheable by CDNs, and testable in CI. Then add `Accept: text/markdown` negotiation on the canonical URL as a convenience, with `Vary: Accept` set so caches do not serve the wrong representation.

### Does a .md variant create duplicate content problems for SEO?

Point the Markdown variant at the HTML page as canonical (via an `X-Robots-Tag`/`Link: rel="canonical"` header, since Markdown has no head element) and reference the pair with `rel="alternate"` from the HTML. Treat the HTML as the indexable representation and the Markdown as an alternate format of the same resource.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [llms.txt vs robots.txt vs sitemap.xml: Which File Does What](https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md): The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [How AI Search Engines Choose Sources (And What You Can Actually Influence)](https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md): What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [What to Charge AI Crawlers: Pricing Models for Machine Buyers](https://changegamer.ai/articles/what-to-charge-ai-crawlers.md): Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [JSON API Design for AI Agents: Endpoints They Prefer Over Scraping](https://changegamer.ai/articles/json-api-design-for-agents.md): How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.
- [Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions](https://changegamer.ai/articles/measuring-ai-agent-traffic.md): Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.
- [Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them](https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md): A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

## Reference resources

- https://changegamer.ai/resources/data-formats.md
- https://changegamer.ai/resources/llms-txt-explained.md
- https://changegamer.ai/resources/json-api.md
- https://changegamer.ai/resources/getting-started.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt