# JSON API Design for AI Agents: Endpoints They Prefer Over Scraping

> How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.

Guide: The agent-ready web — part 9
Published: 2026-07-26 · Updated: 2026-07-26 · 1288 words
Canonical: https://changegamer.ai/articles/json-api-design-for-agents
JSON: https://changegamer.ai/api/articles/json-api-design-for-agents.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- Publish one discovery endpoint that lists every other endpoint with its content type and auth requirement. Without it, an agent guesses URLs.
- Give every collection a cheap freshness signal — a stats or version endpoint — so pollers stop re-fetching everything to learn nothing changed.
- Bulk beats pagination for corpora. One NDJSON export saves an agent hundreds of round trips and saves you the traffic.
- Errors must be machine-actionable: a stable error code, what to do next, and where the terms are. `{"error":"not_found"}` with no next step wastes a retry budget.
- Ship an OpenAPI description. Toolchains generate agent tools from it, which turns your API into something an agent can call without bespoke integration code.

---

An agent will scrape your HTML if it has to. It would much rather call an endpoint — and so would you, since a JSON response costs a fraction of a rendered page in both bandwidth and ambiguity. This is the API chapter of [the agent-ready website](/articles/agent-ready-website).

## Start with discovery

The endpoint most sites are missing is the one that lists the others. Without it, an agent either reads your documentation (expensive, and only if it finds it) or guesses paths (wasteful, and mostly 404s).

```json
// GET /api/index.json
{
  "site": "Example",
  "url": "https://example.com",
  "updated": "2026-07-26",
  "endpoints": [
    { "url": "https://example.com/api/items.json", "type": "application/json",
      "auth": "none", "description": "Index of every item: id, title, updated, urls." },
    { "url": "https://example.com/api/items/{id}.json", "type": "application/json",
      "auth": "none", "description": "One item with full body." },
    { "url": "https://example.com/api/corpus.jsonl", "type": "application/x-ndjson",
      "auth": "none", "description": "Every item, one JSON object per line." },
    { "url": "https://example.com/api/stats.json", "type": "application/json",
      "auth": "none", "description": "Counts and newest-update date. Poll this, not the corpus." },
    { "url": "https://example.com/api/openapi.json", "type": "application/json",
      "auth": "none", "description": "OpenAPI 3.1 description of every endpoint." }
  ]
}
```

Three fields do the work: `type` tells a client how to parse without sniffing, `auth` tells it whether to expect a 401/402 before it wastes a request, and `description` tells it whether the endpoint answers its question at all. Link this file from your llms.txt and you have closed the discovery loop — see [how to write an llms.txt file](/articles/how-to-write-an-llms-txt-file).

## Shapes that survive contact with an agent

### Every item carries its own URLs

An agent that fetches your index should not have to construct URLs by string concatenation. Emit them:

```json
{
  "slug": "agent-traffic",
  "title": "What our logs show about AI crawlers",
  "description": "Server-side measurements of AI crawler behaviour over 90 days.",
  "updated": "2026-07-20",
  "html": "https://example.com/blog/agent-traffic",
  "markdown": "https://example.com/blog/agent-traffic.md",
  "json": "https://example.com/api/items/agent-traffic.json",
  "premium": false
}
```

This one convention removes a whole class of integration bug, and it makes the [Markdown variants](/articles/serving-markdown-variants-to-ai-agents) you shipped actually discoverable from the data path.

### Dates are ISO, always

`"updated": "2026-07-20"` or a full RFC 3339 timestamp. Never a localised string, never a relative "3 days ago", never a Unix integer without documenting the unit. Agents compare dates; ambiguity here produces silent staleness bugs on their side and support tickets on yours.

### Flat beats clever

Deeply nested envelopes (`{data:{attributes:{…}}}`) cost tokens on every single record and buy nothing for a read-only content API. Keep records flat, name fields for what they contain, and put collection metadata in a sibling key rather than wrapping every item.

### Include the relationship graph

If your items relate to each other, say so in the item payload — `related: ["slug-a","slug-b"]`. An agent that gets relationships for free stops trying to infer them from your HTML navigation, and your internal-link structure becomes usable by machines. The shape this site publishes, including its related graph, is documented in [JSON API for agents](/resources/json-api).

## Freshness: give pollers something cheap

Agents poll. If the only way to learn whether anything changed is to fetch the whole corpus, they will fetch the whole corpus — repeatedly. Publish a small endpoint whose only job is to answer "should I re-fetch?":

```json
// GET /api/stats.json  — a few hundred bytes
{
  "count": 78,
  "corpus_version": "1.42.0",
  "newest_update": "2026-07-24",
  "by_category": { "guide": 41, "reference": 30, "policy": 7 }
}
```

Then support the HTTP machinery that makes polite polling possible: `ETag` and `Last-Modified` on every endpoint, honouring `If-None-Match` and `If-Modified-Since` with a 304. A well-built agent will use them, and your bandwidth bill reflects it.

Feeds do the same job for change-driven consumers: Atom or JSON Feed of additions and updates, newest first. Cheap to serve, universally understood.

## Bulk export: the endpoint agents want most

For anything corpus-shaped, one NDJSON file beats paginated JSON on every axis:

```
# GET /api/corpus.jsonl
{"record":"meta","corpus_version":"1.42.0","count":78,"generated":"2026-07-26"}
{"slug":"agent-traffic","title":"…","updated":"2026-07-20","text":"# What our logs…"}
{"slug":"crawler-policy","title":"…","updated":"2026-07-18","text":"# Which crawlers…"}
```

Design notes:

- **First line is a meta record** with a version and a count. A consumer can validate it received the whole file, and can tell whether it already has this version.
- **One record per line, no trailing commas, no outer array.** That is the entire point: line-oriented parsing without loading the file into memory.
- **Publish the schema.** A JSON Schema document for the record shape lets a consumer validate instead of guessing which fields are optional.
- **If some records are paid,** include the record with its body set to `null` plus an access pointer (price, checkout URL). A gap in the file is indistinguishable from a bug; an explicit stub is a sales pitch. That contract is described in [access and pricing for crawlers](/resources/access-and-pricing).

## Errors an agent can act on

The difference between a usable and an unusable API is mostly in the error bodies. An agent has a retry budget and no intuition:

```json
// 404
{ "error": "not_found", "slug": "nonexistent",
  "hint": "Fetch https://example.com/api/items.json for the full list of valid slugs." }

// 429
{ "error": "rate_limited", "retry_after_seconds": 30,
  "limit": "60 requests/minute", "bulk_alternative": "https://example.com/api/corpus.jsonl" }

// 402
{ "error": "payment_required", "price": 5, "currency": "EUR",
  "payment_url": "https://buy.example.com/key",
  "how_to_pay": "Buy a key, then retry with Authorization: Bearer <key>.",
  "terms": "https://example.com/terms.md" }
```

Every one has a stable machine code, the parameters needed to comply, and a next action. On 429s, send a real `Retry-After` header as well — the agent-side contract for this is in [handling LLM rate limits and retries](/resources/handling-rate-limits-and-retries), and pointing rate-limited clients at a bulk endpoint converts a fight into a cheaper fetch for both sides. The 402 shape is worked through in [implementing an HTTP 402 paywall](/articles/http-402-paywall-implementation).

## Ship an OpenAPI description

An OpenAPI 3.1 document at a stable path is what turns your endpoints into *tools*. Toolchains generate callable functions from it, so an agent can use your API without anyone writing an integration. It also forces you to notice inconsistencies in your own parameter naming, which is a free design review.

Keep it generated from the same route definitions that serve traffic. A hand-written spec describes an API you used to have. The consumer-side view of why precise schemas matter — and what breaks without them — is [reliable tool calling and structured outputs](/resources/reliable-tool-calling).

## Operational rules

1. **GET only, unless you truly need writes.** Read-only endpoints are cacheable, safe to retry, and trivially safe to expose.
2. **Cache free responses at the edge** and set honest `Cache-Control`. Never cache paid or personalised responses in a shared cache.
3. **CORS open on public read endpoints.** Browser-based agents exist; a missing `Access-Control-Allow-Origin` blocks them for no benefit.
4. **Stable paths.** A renamed endpoint is a broken integration in someone's production system. Keep the old path resolvable.
5. **No client-side-only data.** If a value exists only after JavaScript runs, publish it — that is the whole reason agents scrape.
6. **Document limits honestly:** rate limits, page sizes, maximum export size. An agent that knows your limits respects them; one that discovers them by being blocked stops trying.

Streaming is worth a mention for long responses: for generated or very large payloads, incremental delivery changes the client's experience significantly, and the trade-offs are covered in [streaming responses for agents](/resources/streaming-for-agents).

## The smallest viable set

If you build only five endpoints, build these:

| Endpoint | Purpose |
|---|---|
| `/api/index.json` | What endpoints exist, with types and auth |
| `/api/items.json` | Index of everything, with per-item URLs |
| `/api/items/{id}.json` | One item, complete |
| `/api/stats.json` | Freshness and size, cheap to poll |
| `/api/corpus.jsonl` | Bulk export, one record per line |

That set answers every question an agent normally scrapes for, costs a static build step to produce, and is cheaper to serve than the HTML it replaces.

## Frequently asked questions

### Do I need a full REST API for agents?

No. Most content sites need a handful of read-only GET endpoints: an index of what exists, a per-item endpoint, a bulk export, and a freshness signal. Write paths, authentication and versioning complexity are only needed if agents actually change state.

### Should agent endpoints be versioned?

The shape should be stable enough that you rarely need to break it, and additive changes (new fields) should be safe by policy. When you must break something, publish the new shape at a new path and keep the old one resolvable — an agent caches your response shape in its code, not in its memory.

### JSON or NDJSON?

JSON for single objects and small collections, NDJSON for anything an agent will stream or index line by line — corpora, logs, exports. NDJSON is also far friendlier to memory-constrained pipelines, since each line parses independently.

### How do agents discover these endpoints?

Three places, all cheap: a link list in your llms.txt, a discovery index at a predictable path such as `/api/index.json`, and `Link` headers on your HTML pages. Do not rely on a human reading your docs page.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [llms.txt vs robots.txt vs sitemap.xml: Which File Does What](https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md): The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility](https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md): How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.
- [How AI Search Engines Choose Sources (And What You Can Actually Influence)](https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md): What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [What to Charge AI Crawlers: Pricing Models for Machine Buyers](https://changegamer.ai/articles/what-to-charge-ai-crawlers.md): Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions](https://changegamer.ai/articles/measuring-ai-agent-traffic.md): Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.
- [Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them](https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md): A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

## Reference resources

- https://changegamer.ai/resources/json-api.md
- https://changegamer.ai/resources/data-formats.md
- https://changegamer.ai/resources/reliable-tool-calling.md
- https://changegamer.ai/resources/handling-rate-limits-and-retries.md
- https://changegamer.ai/resources/streaming-for-agents.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt