← All guides · The agent-ready web

The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization

Pillar guide · 4,160 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.

For twenty years the web was written for people, with search engines as the intermediary that decided who saw it. That intermediary is being replaced. The client on the other end of an increasing share of your requests is not a person with a browser — it is a model with a task, a token budget, and no tolerance for anything that is not content.

The scale of the shift is now measurable rather than anecdotal: Cloudflare reported that automated systems drove 57.5% of HTTP requests against 42.5% human in mid-2026, the first such crossover it has recorded. Whatever share of that traffic is useful to you, it is not something you can treat as noise.

This guide is the operator side of that shift. It is not about writing content that models like. It is about the technical and commercial architecture that decides whether an agent can get your content at all, whether it can use it once fetched, whether you control what happens to it afterwards, and whether you get paid. Each section links to a focused deep-dive; this page is the map.

The four jobs of an agent-ready site

Everything worth doing falls into four jobs, in strict dependency order. Skipping ahead is the most common and most expensive mistake — a beautifully structured llms.txt behind a WAF rule that 403s GPTBot is a file nobody will ever read.

The order matters because each job depends on the one before it. Control without fetchability is theatre. Payment without readability sells something the buyer cannot consume.

Job 1 — Fetchable: the layer that silently breaks everything

Start here, always. The most common reason a site is invisible to AI agents is not missing metadata; it is an edge rule nobody remembers enabling.

This is not hypothetical. On this site, Cloudflare's managed "Manage AI bots" rule was silently returning 403 to GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, CCBot and Google-CloudVertexBot — including on / and /sitemap.xml — while robots.txt explicitly allowed every one of them. Separately, Cloudflare's Browser Integrity Check, which is on by default, 403'd any client without standard browser headers and in the process broke Google Search Console's sitemap fetch. Both problems were invisible in the dashboard's summary view and both were found only by asking the origin directly.

Verify, do not assume

The check takes thirty seconds per crawler and is the highest-value thing in this guide:

# Does the origin answer a crawler UA with 200?
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://example.com/ | head -1

curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://anthropic.com/aup)" \
  https://example.com/ | head -1

# And the surfaces agents actually depend on
for p in / /robots.txt /sitemap.xml /llms.txt; do
  printf "%s " "$p"
  curl -sI -A "PerplexityBot/1.0" "https://example.com$p" | head -1
done

Anything other than 200 (or a deliberate 402/404) is a bug in your access policy, not a crawler problem. A 403 here means an edge rule is overriding the policy you published — and because robots.txt is fetched over the same path, the crawler may never even learn what your policy is.

The rest of the fetchability checklist

• Status codes must be honest. Soft 404s that return 200 with an error page poison agent pipelines far worse than they poison search indexes, because the agent will confidently quote the error page. Gone means 410 or 404; moved means 301; paywalled means 402 or 401, not 200-with-a-teaser.
• Redirect chains cost real money. Every hop is a round trip against a token and time budget. Canonicalise once: pick trailing-slash or no-trailing-slash and serve that URL directly.
• Rate limits should be graduated, not binary. A crawler that gets 429 with a Retry-After header will come back politely; one that gets a hard block disappears. Well-behaved agents implement backoff — see handling rate limits and retries for what the other side of that contract looks like.
• Do not require JavaScript for primary content. Several crawlers do not execute it at all, and those that do pay a rendering cost that makes your page a low-priority fetch. If your content only exists after hydration, it effectively does not exist. This is the first of the failure modes catalogued in why AI agents can't read your site.
• Do not gate content behind cookie walls or interstitials. An agent has no consent to give and no button to click.

Job 2 — Readable: content a machine can extract in one hop

Once a client can reach you, the question becomes cost. An agent working a task has a token budget; the cheaper your content is to parse, the more of it gets used. Three formats, in ascending order of effort and value.

Semantic HTML is the floor, not the ceiling

Real <article>, <h1>–<h3>, <table> with <th scope>, and <time datetime> cost nothing and survive every extraction pipeline. Div soup with visual-only hierarchy forces the model to infer structure, and inference is where hallucination enters. If you do one thing for readability, make your headings a real outline.

Markdown variants: the highest-value cheap win

Serving a Markdown twin of every content page — /page and /page.md — removes navigation, scripts, styling and ads from what the model sees, typically cutting the token cost of a page by a large multiple with zero loss of meaning. Advertise it from the HTML page so it is discoverable:

<link rel="alternate" type="text/markdown" href="/page.md">

The mechanics, including content negotiation and the pitfalls of generating Markdown from rendered HTML rather than from source, are in serving Markdown variants to AI agents. For the wider format question — Markdown vs JSON vs JSONL vs plain text, and which one an agent should prefer for which job — the reference is data formats and schema.

Discovery files: llms.txt and friends

/llms.txt is a curated, Markdown index of your site written for models: an H1, an optional blockquote summary, and H2 sections of annotated links. It was proposed by Jeremy Howard (Answer.AI) on 3 September 2024 and remains a community convention rather than a ratified standard — useful, cheap, and not a ranking factor. The companion /llms-full.txt inlines the whole corpus for agents that prefer one large fetch.

Three files, three different jobs, and they are routinely confused — the comparison is in llms.txt vs robots.txt vs sitemap.xml, and the exact file format, including the special ## Optional section that agents may skip under context pressure, is in the llms.txt convention explained. If you are writing one from scratch, how to write an llms.txt file is the step-by-step with a template.

Two adjacent conventions are worth knowing: AGENTS.md, which tells coding agents how to work inside a repository rather than how to read a site (AGENTS.md explained), and NLWeb, a proposal for turning a site into a conversational endpoint (NLWeb explained).

Structured data, feeds and APIs

Schema.org markup remains the most widely consumed machine-readable layer on the web, but only a handful of types earn their keep for agent consumers — and some of the effort sites spend on markup would be better spent on a JSON endpoint. Which types actually pay off is in structured data for AI agents.

For anything an agent might want repeatedly — an index, a price list, a changelog — a JSON endpoint beats every scraping strategy on both sides: cheaper for the agent, cheaper for you, and versionable. Design guidance is in JSON API design for agents, with the shape this site publishes documented in JSON API for agents. Atom or JSON Feed on top of that gives pollers a cheap "what changed" signal so they stop re-fetching everything.

MCP: content as a tool call

The Model Context Protocol turns a site into a set of callable tools rather than a set of pages to scrape. For a content business it is a distribution channel: an agent that connects to your MCP server has your catalogue in its context without crawling anything. The strategic case, and the honest limits, are in running an MCP server as a distribution channel; how agents find and judge servers is in finding and evaluating MCP servers, and the implementation path is building an MCP server.

Job 3 — Controllable: say what you mean, in the layer that means it

Control is a stack of four layers with genuinely different force. Sites get this wrong by expecting the weak layers to do the strong layers' work.

Per-agent robots.txt is a business decision, not a checkbox

The token table matters because the tokens do different things. GPTBot collects training data; OAI-SearchBot indexes for ChatGPT Search; ChatGPT-User is a live, user-triggered fetch — and since an OpenAI documentation update in December 2025, only the first two are listed as robots.txt-compliant. Anthropic's ClaudeBot, Claude-SearchBot and Claude-User split the same three ways. Perplexity-User ignores robots.txt by design. Google-Extended and Applebot-Extended are not crawlers at all: they are training opt-out tokens whose fetching is still done by Googlebot and Applebot, which is exactly why disallowing them costs you no search visibility.

The full token table with vendor, purpose and compliance status is maintained in AI crawler policy: robots.txt and user-agents. The decision of which of them to allow — and the business logic behind each answer — is should you block AI crawlers?.

Usage signals: the cheapest layer to add

Cloudflare announced the Content Signals Policy on 24 September 2025: three directives — search, ai-input, ai-train — expressed inside a robots.txt user-agent block as a comment, so unaware parsers ignore them safely. Cloudflare defaulted the policy on for over 3.8 million domains on its managed robots.txt feature, at search=yes, ai-train=no. Cloudflare itself is clear that these are preferences rather than a technical block, while noting they "might have legal rights in various jurisdictions" — citing the EU text-and-data-mining opt-out in Directive 2019/790. The IETF chartered an AI Preferences (AIPREF) Working Group in 2025 to standardise the same idea; as of mid-2026 it is still in progress, not an RFC. Details in content signals explained.

Licensing: from preference to terms

RSL (Really Simple Licensing) is a machine-readable licence document — typically an XML file such as /license.xml, discovered via a License: directive in robots.txt — that states permitted uses and compensation requirements. It is a licensing layer, not a settlement layer: it declares terms, it does not collect money. Provenance standards sit alongside it: C2PA Content Credentials cryptographically bind assertions about origin to a file (C2PA content credentials). If your content is the product, licensing content for AI training walks through what to publish and what to expect.

A newer strand of work aims at controlling agents rather than crawlers — declaring which automated actions a site permits. That is the subject of the agent control specification, and it matters more as browser-driving agents proliferate (agentic browsers).

Job 4 — Payable: pricing a machine buyer

If agents consume your content and you want revenue from it, you need a path a machine can complete alone. Human checkout is not that path: an agent cannot pass an email verification loop or read a confirmation inbox.

Three mechanisms are live, and they are not competitors so much as different layers:

1. Self-hosted HTTP 402. Your origin returns 402 Payment Required with a JSON body a machine can parse: price, payment URL, how to retry, link to terms and licence. The agent buys a key and retries with Authorization: Bearer …. Maximum control, you own billing and support. This is the mechanism this site runs — the wire contract is in paying for access: the HTTP 402 flow, and the implementation walkthrough is implementing an HTTP 402 paywall.
2. Cloudflare Pay Per Crawl. A CDN-layer 402 marketplace: Cloudflare intercepts crawler requests, handles price negotiation via crawler-max-price, acts as merchant of record and remits payouts. It remained in closed/private beta as of July 2026. On 1 July 2026 Cloudflare announced a broader Monetization Gateway (waitlist stage) generalising the same pattern from crawler content to any protected resource, including APIs and MCP tools.
3. x402. An open protocol settling on-chain in stablecoins (mainly USDC) across multiple network families, governed by the Linux Foundation's x402 Foundation, which grew from 22 founding members in April 2026 to 40 — including premier members Visa, Mastercard and Ripple — at its operational launch on 14 July 2026. Adoption jumped roughly tenfold after Apify put 20,000-plus Actors on the protocol. It is the only rail where a wallet-holding agent can pay end-to-end with no account and no human.

A separate class of standards — ACP (Stripe/OpenAI) and AP2 (Google, governed by the FIDO Alliance) — solves a different problem: letting an agent check out with its owner's money and consent. Useful if you sell products to agents acting for humans; irrelevant if you are pricing content access. The comparison is in agentic payment protocols.

The hard part is not the plumbing, it is the number. Per-crawl pricing, corpus licensing and subscription keys behave very differently at agent scale; what to charge AI crawlers works through the arithmetic and the failure modes, including the one that catches everyone: pricing before you have demand data.

Measurement: if you cannot see agents, you cannot decide anything

Client-side analytics miss almost all agent traffic, because most agents do not execute your JavaScript. Everything useful is server-side. The minimum viable telemetry is one row per request with: timestamp, path, user agent, status code, and outcome class (served, blocked, 402, upgrade-required). That is enough to answer the only four questions that drive decisions:

• Which agents visit, and are they real? Vendor UA strings are trivially spoofed; every major vendor publishes IP ranges, and verification is the only way to know.
• What do they fetch? If demand concentrates on a handful of URLs, those are your product. If it is spread thin, your corpus is the product, not any page.
• What do they get? A wall of 403s means your edge is overriding your policy. A wall of 402s with no purchases means your price or your payment path is wrong.
• Does agent traffic convert? Citations in AI answers, referral clicks from answer engines, and — if you sell access — direct machine purchases.

The instrumentation recipe, including how to sample without losing the signal and how to attribute answer-engine referrals, is in measuring AI agent traffic. What answer engines appear to reward once they can read you is in how AI search engines choose sources.

What transfers from classic SEO, and what does not

Teams arriving from an SEO background bring habits that are half useful. The split is sharp enough to be worth naming, because the useless half consumes most of the budget.

The uncomfortable summary: the technical hygiene half of SEO is table stakes for agent-readiness, and the persuasion half is close to irrelevant. What replaces persuasion is being verifiable — dated claims, named sources, explicit terms — because a model deciding whether to cite you is, in effect, assessing risk.

A worked example: how this site is built

It helps to see the four jobs in one architecture. This site is a static build where a single source-of-truth data file generates every surface, which is what keeps the machine variants from drifting away from the HTML:

• Fetchable. Static assets behind a Worker; index files (/robots.txt, /llms.txt, /sitemap-index.xml, /license.xml) bypass the gate entirely and can never be paywalled by code. Crawler access is re-verified with spoofed-UA curl after every edge change.
• Readable. Every resource ships three ways from one source: an HTML page, a Markdown twin at /resources/<slug>.md, and JSON at /api/resources/<slug>.json. The whole corpus is available as one file (/llms-full.txt), as NDJSON (/api/corpus.jsonl), and as tool calls over MCP at /mcp.
• Controllable. robots.txt is written per user-agent token, and an RSL licence document at /license.xml states terms in machine-readable form.
• Payable. Premium resources return 402 Payment Required with a JSON body carrying price, checkout URL, retry instructions, and links to both terms and licence — plus Link headers advertising the payment manifest. An agent can go from "blocked" to "paid and reading" without a human.

The generalisable lesson is not the stack, it is the invariant: one source of truth, many rendered surfaces. Sites that hand-maintain a Markdown copy or an llms.txt file alongside their CMS end up shipping two versions of the truth, and the machine-readable one rots first.

A 30-day implementation plan

Ordered by value per hour, not by novelty. Weeks are calendar guidance for a small team; a determined engineer does week 1 and 2 in three days.

Week 1 — Make sure you are reachable

1. Run the crawler-UA curl matrix above against /, /robots.txt, /sitemap.xml, and your three most valuable pages. Fix every unexpected 403 at the WAF or bot-management layer.
2. Turn off browser-integrity-style checks on content paths, or scope them to authenticated routes only.
3. Audit status codes: no soft 404s, no 200-with-error-page, no redirect chains longer than one hop on canonical URLs.
4. Confirm primary content is present in the raw HTML response (curl -s URL | grep -c "<your first paragraph>"), not injected after hydration.

Week 2 — Make yourself cheap to read

5. Ship Markdown variants for your main content type and advertise them with <link rel="alternate" type="text/markdown">.
6. Fix heading hierarchy and convert visual tables to real <table> markup.
7. Add or correct the two or three schema.org types that actually apply.
8. Publish /llms.txt — an H1, a one-paragraph summary, and annotated link lists pointing at your Markdown variants.

Week 3 — Say what you mean

9. Rewrite robots.txt per user-agent token, deliberately: search-indexing bots, live user fetchers, and training crawlers are three separate decisions.
10. Add Content Signals (search=…, ai-train=…) to match the policy you actually want.
11. If your content has licensing value, publish an RSL document and reference it with a License: directive.
12. Re-run the week 1 verification matrix — policy changes routinely break access.

Week 4 — Instrument, then decide about money

13. Log every request server-side with UA, path, status and outcome. Keep 30 days.
14. Build one dashboard answering the four questions above.
15. Only now consider a payment rail — and pick it from your data: concentrated demand on a few URLs argues for per-resource 402; broad shallow crawling argues for corpus licensing or Pay Per Crawl.
16. Publish machine-readable terms (/api/payment.json-style) next to whatever rail you choose, so an agent can discover the price without guessing.

What to skip

Being honest about the low-value end saves more time than any checklist item:

• Do not buy "AI SEO" tooling before your logs show agent traffic. Instrumentation first; the tooling market is currently ahead of the evidence.
• Do not write content specifically to flatter models. The observable signals — clear structure, dated claims, sources — are the same signals that serve human readers.
• Do not chase every emerging standard. Several of the mechanisms in this guide are drafts or betas; treat anything without a stable spec as a cheap experiment, not a migration.
• Do not paywall your terms. If your 402 body links to a licence page, that page must be free, or your payment flow is a dead end. The same applies to robots.txt and llms.txt: index and policy files should never be gated.

Where this is going

Three trend lines are worth tracking, because each changes the architecture rather than the tactics.

Preferences are becoming standards. The IETF AIPREF work is the first serious attempt to give usage preferences a spec-grade vocabulary. If it lands, "no training" stops being a Cloudflare feature and becomes a protocol.

Payment is becoming ambient. With Visa, Mastercard and Ripple joining the x402 Foundation as premier members and Cloudflare generalising per-crawl payment into a broader monetization gateway, the assumption that a buyer must be a human with an account is eroding. Sites that publish machine-readable prices will be discoverable by buyers that never visit a homepage.

Agents are becoming clients, not just crawlers. Browser-driving agents and MCP clients do not read pages so much as operate them. That shifts the question from "can it parse my article" to "can it complete my flow" — and sites that treat agent access as a first-class interface, rather than an SEO afterthought, will be the ones that work. The pattern catalogue for that world is in web data and scraping for agents and agentic browsers.

The cluster: every sub-topic in depth

This pillar is deliberately a map. Each article below owns one decision and goes deeper than a section here can:

Discovery and formats

• llms.txt vs robots.txt vs sitemap.xml — three files, three jobs, one common confusion.
• How to write an llms.txt file — format, template, validation, maintenance.
• Serving Markdown variants to AI agents — the cheapest token-cost win available.
• Structured data for AI agents — which schema.org types earn their keep.
• JSON API design for agents — endpoints agents prefer over scraping.

Visibility and control

• How AI search engines choose sources — what appears to drive citation.
• Should you block AI crawlers? — a decision framework by business model.
• Why AI agents can't read your site — the twelve recurring failure modes.
• Measuring AI agent traffic — server-side telemetry that answers real questions.

Money and rights

• What to charge AI crawlers — pricing models and the arithmetic behind them.
• Implementing an HTTP 402 paywall — a machine-payable gate, end to end.
• Licensing content for AI training — RSL, terms, provenance.
• Running an MCP server as a distribution channel — content as tool calls.

Sources and further reading

Every factual claim above is carried, with its primary source, by a reference resource in this site's corpus:

• Crawler tokens, compliance status and the WAF layer: /resources/ai-crawler-policy
• llms.txt format and provenance: /resources/llms-txt-explained
• Content Signals and IETF AIPREF: /resources/content-signals-explained
• 402, Pay Per Crawl, x402, ACP/AP2, RSL: /resources/agentic-payment-protocols
• The 402 wire contract: /resources/paying-for-access-402
• Formats and schema for machine consumers: /resources/data-formats

Agents: every article in this cluster has a Markdown variant at /articles/<slug>.md, and the whole cluster is indexed as JSON at /api/articles.json.

#ai-visibility #agents #crawlers #llms.txt #monetization #seo