← All guides · The agent-ready web

Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility

Part 3 of The agent-ready web · 1,091 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.

Part of the The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization guide.

Of everything in the agent-ready website, this is the change with the best ratio of effort to effect: publish a clean Markdown version of each content page and tell clients it exists.

The reason is arithmetic. A model does not read your page, it pays for it. Every token spent on your navigation, your cookie notice, your inline SVG icons and your analytics snippet is a token not spent on your content — and when the budget runs out, the model summarises from whatever it managed to load. Markdown deletes the overhead without deleting anything that carries meaning.

What actually gets removed

Take a typical article page and strip it to Markdown. What disappears:

• header and footer navigation, breadcrumb bars, sidebars, related-post rails
• <script> and <style> blocks, inline critical CSS, JSON-LD you already expose elsewhere
• cookie/consent markup, newsletter modals, share buttons, ad slots
• wrapper <div> scaffolding, utility classes, data attributes

What survives: headings, paragraphs, lists, tables, code blocks, links, emphasis. In other words, the document. The gain varies with how heavy your template is, and on a modern JS-framework page it can be most of the payload.

There is a second, subtler gain: unambiguous structure. In HTML a model has to infer that a bold <div> is a heading. In Markdown a heading is ##. Inference is where extraction errors — and therefore misquotations of your content — come from. For the broader format comparison (Markdown vs JSON vs JSONL vs plain text, and when each is the right answer) see data formats and schema.

The URL pattern

Use a suffix on the canonical path:

https://example.com/blog/agent-traffic          → HTML
https://example.com/blog/agent-traffic.md       → Markdown

Why a suffix beats the alternatives:

Serve it as Content-Type: text/markdown; charset=utf-8. Do not serve Markdown as text/plain if you can avoid it, and never as text/html.

Make it discoverable

A variant nobody can find is wasted work. Announce it in three places:

<!-- 1. In the HTML head of the canonical page -->
<link rel="alternate" type="text/markdown" href="/blog/agent-traffic.md">

# 2. As a response header, so a HEAD request is enough to learn about it
Link: </blog/agent-traffic.md>; rel="alternate"; type="text/markdown"

# 3. In llms.txt, where the link target IS the .md variant
- [Agent traffic](https://example.com/blog/agent-traffic.md): what our logs show about AI crawlers

The third is the one that matters most in practice, because an agent that starts at llms.txt then never touches your HTML at all.

Generate from source, not from HTML

This is the trap. The tempting implementation is a route that fetches the rendered page and runs it through an HTML-to-Markdown converter. It is fast to build and it reintroduces exactly what you were trying to remove: the converter faithfully preserves your navigation as a bullet list, your footer as a link soup, and your cookie banner as a paragraph.

Generate from the content source instead:

// Astro-style: one entry, two rendered surfaces
// pages/blog/[slug].astro  → HTML (layout + nav + styles)
// pages/blog/[slug].md.ts  → Markdown (body only)
export const GET = ({ props }) => {
  const p = props.post;
  const out = [
    `# ${p.title}`, "",
    `> ${p.description}`, "",
    `Updated: ${p.updated} · Canonical: ${SITE.url}/blog/${p.slug}`, "",
    p.body,                       // the SAME markdown the HTML page renders
  ].join("\n");
  return new Response(out, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Link": `<${SITE.url}/blog/${p.slug}>; rel="canonical"`,
    },
  });
};

If your CMS stores rich text rather than Markdown, convert once at ingest or at build from the structured representation — not from the final DOM.

What to include in the variant

Minimal, but not naked. A model that fetches only the .md should still know what it has:

1. Title as H1.
2. One-line summary (blockquote or plain line).
3. Provenance line: last-updated date and the canonical HTML URL. This is what lets an agent cite you correctly rather than citing "a webpage".
4. The body, headings intact.
5. Optionally a short list of related URLs at the end — internal linking works on machine readers too.

What to leave out: navigation, calls to action repeated from the template, tracking parameters on links, and anything you would not want quoted verbatim.

Canonicalisation and indexing

Markdown has no <head>, so canonical signals have to travel in headers:

Content-Type: text/markdown; charset=utf-8
Link: <https://example.com/blog/agent-traffic>; rel="canonical"

Treat the HTML page as the indexable representation, keep the pair explicitly linked with rel="alternate", and include only the HTML URL in your sitemap. Some sites additionally set X-Robots-Tag: noindex on .md variants; that is defensible if you are worried about split signals, but it also tells crawlers not to index the cheapest version of your content — decide deliberately rather than by default.

Verify it works

# Right content type, no HTML leakage
curl -sI https://example.com/blog/agent-traffic.md | grep -i content-type
curl -s  https://example.com/blog/agent-traffic.md | grep -c "<div"   # expect 0

# Same content, far smaller payload
for u in /blog/agent-traffic /blog/agent-traffic.md; do
  printf "%-28s %s bytes\n" "$u" \
    "$(curl -so /dev/null -w "%{size_download}" "https://example.com$u")"
done

# Discoverable from the HTML page
curl -s https://example.com/blog/agent-traffic | grep "text/markdown"

Then add two CI assertions: every HTML page has a resolvable .md twin, and every .md twin contains the page's first heading. That is enough to catch the drift that otherwise creeps in during template refactors.

When Markdown is the wrong answer

Markdown is for documents. For everything that is really data — price lists, catalogues, availability, changelogs, search results — a JSON endpoint is better on both sides: smaller, unambiguous, and versionable. Publishing both is normal; the rule of thumb is prose in Markdown, records in JSON. See JSON API design for agents and the shape this site publishes in JSON API for agents.

And for very large corpora, one file per page is not always what an agent wants: a single concatenated /llms-full.txt, or NDJSON of the whole corpus, saves it hundreds of round trips. The onboarding contract that explains all of these variants to an agent in one place is getting started for agents.

#markdown #formats #agents #ai-visibility #tokens