Do I need a full REST API for agents?

No. Most content sites need a handful of read-only GET endpoints: an index of what exists, a per-item endpoint, a bulk export, and a freshness signal. Write paths, authentication and versioning complexity are only needed if agents actually change state.

Should agent endpoints be versioned?

The shape should be stable enough that you rarely need to break it, and additive changes (new fields) should be safe by policy. When you must break something, publish the new shape at a new path and keep the old one resolvable — an agent caches your response shape in its code, not in its memory.

JSON for single objects and small collections, NDJSON for anything an agent will stream or index line by line — corpora, logs, exports. NDJSON is also far friendlier to memory-constrained pipelines, since each line parses independently.

How do agents discover these endpoints?

Three places, all cheap: a link list in your llms.txt, a discovery index at a predictable path such as `/api/index.json`, and `Link` headers on your HTML pages. Do not rely on a human reading your docs page.

← All guides · The agent-ready web

JSON API Design for AI Agents: Endpoints They Prefer Over Scraping

Part 9 of The agent-ready web · 1,288 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.

In short

Publish one discovery endpoint that lists every other endpoint with its content type and auth requirement. Without it, an agent guesses URLs.
Give every collection a cheap freshness signal — a stats or version endpoint — so pollers stop re-fetching everything to learn nothing changed.
Bulk beats pagination for corpora. One NDJSON export saves an agent hundreds of round trips and saves you the traffic.
Errors must be machine-actionable: a stable error code, what to do next, and where the terms are. {"error":"not_found"} with no next step wastes a retry budget.
Ship an OpenAPI description. Toolchains generate agent tools from it, which turns your API into something an agent can call without bespoke integration code.

Part of the The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization guide.

An agent will scrape your HTML if it has to. It would much rather call an endpoint — and so would you, since a JSON response costs a fraction of a rendered page in both bandwidth and ambiguity. This is the API chapter of the agent-ready website.

Start with discovery

The endpoint most sites are missing is the one that lists the others. Without it, an agent either reads your documentation (expensive, and only if it finds it) or guesses paths (wasteful, and mostly 404s).

// GET /api/index.json
{
  "site": "Example",
  "url": "https://example.com",
  "updated": "2026-07-26",
  "endpoints": [
    { "url": "https://example.com/api/items.json", "type": "application/json",
      "auth": "none", "description": "Index of every item: id, title, updated, urls." },
    { "url": "https://example.com/api/items/{id}.json", "type": "application/json",
      "auth": "none", "description": "One item with full body." },
    { "url": "https://example.com/api/corpus.jsonl", "type": "application/x-ndjson",
      "auth": "none", "description": "Every item, one JSON object per line." },
    { "url": "https://example.com/api/stats.json", "type": "application/json",
      "auth": "none", "description": "Counts and newest-update date. Poll this, not the corpus." },
    { "url": "https://example.com/api/openapi.json", "type": "application/json",
      "auth": "none", "description": "OpenAPI 3.1 description of every endpoint." }
  ]
}

Three fields do the work: type tells a client how to parse without sniffing, auth tells it whether to expect a 401/402 before it wastes a request, and description tells it whether the endpoint answers its question at all. Link this file from your llms.txt and you have closed the discovery loop — see how to write an llms.txt file.

Shapes that survive contact with an agent

Every item carries its own URLs

An agent that fetches your index should not have to construct URLs by string concatenation. Emit them:

{
  "slug": "agent-traffic",
  "title": "What our logs show about AI crawlers",
  "description": "Server-side measurements of AI crawler behaviour over 90 days.",
  "updated": "2026-07-20",
  "html": "https://example.com/blog/agent-traffic",
  "markdown": "https://example.com/blog/agent-traffic.md",
  "json": "https://example.com/api/items/agent-traffic.json",
  "premium": false
}

This one convention removes a whole class of integration bug, and it makes the Markdown variants you shipped actually discoverable from the data path.

Dates are ISO, always

"updated": "2026-07-20" or a full RFC 3339 timestamp. Never a localised string, never a relative "3 days ago", never a Unix integer without documenting the unit. Agents compare dates; ambiguity here produces silent staleness bugs on their side and support tickets on yours.

Flat beats clever

Deeply nested envelopes ({data:{attributes:{…}}}) cost tokens on every single record and buy nothing for a read-only content API. Keep records flat, name fields for what they contain, and put collection metadata in a sibling key rather than wrapping every item.

Include the relationship graph

If your items relate to each other, say so in the item payload — related: ["slug-a","slug-b"]. An agent that gets relationships for free stops trying to infer them from your HTML navigation, and your internal-link structure becomes usable by machines. The shape this site publishes, including its related graph, is documented in JSON API for agents.

Freshness: give pollers something cheap

Agents poll. If the only way to learn whether anything changed is to fetch the whole corpus, they will fetch the whole corpus — repeatedly. Publish a small endpoint whose only job is to answer "should I re-fetch?":

// GET /api/stats.json  — a few hundred bytes
{
  "count": 78,
  "corpus_version": "1.42.0",
  "newest_update": "2026-07-24",
  "by_category": { "guide": 41, "reference": 30, "policy": 7 }
}

Then support the HTTP machinery that makes polite polling possible: ETag and Last-Modified on every endpoint, honouring If-None-Match and If-Modified-Since with a 304. A well-built agent will use them, and your bandwidth bill reflects it.

Feeds do the same job for change-driven consumers: Atom or JSON Feed of additions and updates, newest first. Cheap to serve, universally understood.

Bulk export: the endpoint agents want most

For anything corpus-shaped, one NDJSON file beats paginated JSON on every axis:

# GET /api/corpus.jsonl
{"record":"meta","corpus_version":"1.42.0","count":78,"generated":"2026-07-26"}
{"slug":"agent-traffic","title":"…","updated":"2026-07-20","text":"# What our logs…"}
{"slug":"crawler-policy","title":"…","updated":"2026-07-18","text":"# Which crawlers…"}

Design notes:

First line is a meta record with a version and a count. A consumer can validate it received the whole file, and can tell whether it already has this version.
One record per line, no trailing commas, no outer array. That is the entire point: line-oriented parsing without loading the file into memory.
Publish the schema. A JSON Schema document for the record shape lets a consumer validate instead of guessing which fields are optional.
If some records are paid, include the record with its body set to null plus an access pointer (price, checkout URL). A gap in the file is indistinguishable from a bug; an explicit stub is a sales pitch. That contract is described in access and pricing for crawlers.

Errors an agent can act on

The difference between a usable and an unusable API is mostly in the error bodies. An agent has a retry budget and no intuition:

// 404
{ "error": "not_found", "slug": "nonexistent",
  "hint": "Fetch https://example.com/api/items.json for the full list of valid slugs." }

// 429
{ "error": "rate_limited", "retry_after_seconds": 30,
  "limit": "60 requests/minute", "bulk_alternative": "https://example.com/api/corpus.jsonl" }

// 402
{ "error": "payment_required", "price": 5, "currency": "EUR",
  "payment_url": "https://buy.example.com/key",
  "how_to_pay": "Buy a key, then retry with Authorization: Bearer <key>.",
  "terms": "https://example.com/terms.md" }

Every one has a stable machine code, the parameters needed to comply, and a next action. On 429s, send a real Retry-After header as well — the agent-side contract for this is in handling LLM rate limits and retries, and pointing rate-limited clients at a bulk endpoint converts a fight into a cheaper fetch for both sides. The 402 shape is worked through in implementing an HTTP 402 paywall.

Ship an OpenAPI description

An OpenAPI 3.1 document at a stable path is what turns your endpoints into tools. Toolchains generate callable functions from it, so an agent can use your API without anyone writing an integration. It also forces you to notice inconsistencies in your own parameter naming, which is a free design review.

Keep it generated from the same route definitions that serve traffic. A hand-written spec describes an API you used to have. The consumer-side view of why precise schemas matter — and what breaks without them — is reliable tool calling and structured outputs.

Operational rules

GET only, unless you truly need writes. Read-only endpoints are cacheable, safe to retry, and trivially safe to expose.
Cache free responses at the edge and set honest Cache-Control. Never cache paid or personalised responses in a shared cache.
CORS open on public read endpoints. Browser-based agents exist; a missing Access-Control-Allow-Origin blocks them for no benefit.
Stable paths. A renamed endpoint is a broken integration in someone's production system. Keep the old path resolvable.
No client-side-only data. If a value exists only after JavaScript runs, publish it — that is the whole reason agents scrape.
Document limits honestly: rate limits, page sizes, maximum export size. An agent that knows your limits respects them; one that discovers them by being blocked stops trying.

Streaming is worth a mention for long responses: for generated or very large payloads, incremental delivery changes the client's experience significantly, and the trade-offs are covered in streaming responses for agents.

The smallest viable set

If you build only five endpoints, build these:

Endpoint	Purpose
`/api/index.json`	What endpoints exist, with types and auth
`/api/items.json`	Index of everything, with per-item URLs
`/api/items/{id}.json`	One item, complete
`/api/stats.json`	Freshness and size, cheap to poll
`/api/corpus.jsonl`	Bulk export, one record per line

That set answers every question an agent normally scrapes for, costs a static build step to produce, and is cheaper to serve than the HTML it replaces.

Frequently asked questions

Do I need a full REST API for agents?: No. Most content sites need a handful of read-only GET endpoints: an index of what exists, a per-item endpoint, a bulk export, and a freshness signal. Write paths, authentication and versioning complexity are only needed if agents actually change state.
Should agent endpoints be versioned?: The shape should be stable enough that you rarely need to break it, and additive changes (new fields) should be safe by policy. When you must break something, publish the new shape at a new path and keep the old one resolvable — an agent caches your response shape in its code, not in its memory.
JSON or NDJSON?: JSON for single objects and small collections, NDJSON for anything an agent will stream or index line by line — corpora, logs, exports. NDJSON is also far friendlier to memory-constrained pipelines, since each line parses independently.
How do agents discover these endpoints?: Three places, all cheap: a link list in your llms.txt, a discovery index at a predictable path such as `/api/index.json`, and `Link` headers on your HTML pages. Do not rely on a human reading your docs page.

#json #api #agents #openapi #formats

Agents: this guide is available as Markdown and JSON; the whole cluster is indexed at /api/articles.json. The reference corpus behind it is at /llms.txt, with licensing at pricing.