Building an MCP Server

Implementation guide for MCP servers: architecture roles, the three server primitives, stdio vs Streamable HTTP transports, official SDKs, server lifecycle, remote-server concerns, testing with MCP Inspector, and publishing to the official registry.

The Model Context Protocol (MCP) gives a host application a uniform way to connect an AI model to external tools, data, and prompt templates. This guide covers what you need to implement a server — from wire format to deployment. For finding and evaluating existing servers see /resources/mcp-server-discovery; for OAuth 2.1 auth on remote servers see /resources/mcp-server-authentication.

## Key facts

- Every MCP deployment involves three roles — a host application, a client managing one connection per server, and the server itself — exchanging JSON-RPC 2.0 messages.
- Servers expose three distinct primitives suited to different jobs: tools for the agent to *do* something, resources for it to *read* subscribable URI-identified data, and prompts for distributing ready-made, parameterized instruction snippets.
- The two standard transports have very different footprints: stdio runs locally as a child process with zero network exposure, while Streamable HTTP runs as a remote endpoint that has to check where each incoming connection claims to originate from, guarding against DNS-rebinding attacks.
- Every session, on either transport, follows the same lifecycle: initialize, initialized, discovery (listing tools/resources/prompts), invocation, and shutdown.
- Remote Streamable HTTP servers carry extra concerns local stdio servers don't: PKCE-based OAuth belongs on that side, a missing session ID leaves the server effectively stateless, and CORS needs to be locked down to known client origins.
- MCP Inspector is the official tool for manually connecting to a server, browsing its primitives, and inspecting the raw JSON-RPC traffic.
- Publishing to the official registry means adding a `server.json` manifest describing the server, then running the `mcp-publisher` CLI to log in and publish.

## Architecture: host, client, server

Three roles exist in every MCP deployment:

- **Host** — the application that owns the model interaction (e.g. Claude for Desktop, a custom agent). It spawns or connects to MCP servers and mediates access.
- **Client** — a component inside the host that manages one connection to one MCP server, running the JSON-RPC 2.0 message exchange.
- **Server** — the process or endpoint you build. It declares capabilities during initialization and handles requests for tools, resources, and prompts.

All messages between client and server are JSON-RPC 2.0 objects: requests carry an `id`, `method`, and optional `params`; responses carry the same `id` plus `result` or `error`. Notifications are one-way (no `id`, no response expected).

## The three server primitives

**Tools** are model-invoked functions. The client calls `tools/list` to discover them; the model selects and calls one via `tools/call`. Each tool has a `name`, `description`, and an `inputSchema` (JSON Schema). Results are returned as typed content blocks (text, image, audio, resource link). Use tools when the agent needs to *do* something — query an API, run a calculation, write to a database.

**Resources** are read-only data the client can subscribe to: files, database rows, live feeds. They are identified by URI. Use resources when the agent needs to *read* structured data that may change over time.

**Prompts** are named, parameterised prompt templates served by the server. The client fetches them via `prompts/get` and injects them into the conversation. Use prompts to share curated instruction fragments without embedding them in the client.

## Transports: stdio and Streamable HTTP

The current MCP spec (stable revision: **2025-11-25**; treat as a snapshot, verify at modelcontextprotocol.io) defines two standard transports.

**stdio (local)** — the host launches the server as a child process. JSON-RPC messages are exchanged over stdin/stdout, delimited by newlines. The server MUST NOT write anything to stdout except valid MCP messages; use stderr for logs. stdio carries zero network exposure and is the right choice for local tools running on the same machine.

**Streamable HTTP (remote)** — the server runs as an independent network process. All client messages are HTTP POST requests to a single MCP endpoint; the server may respond with `application/json` (single response) or `text/event-stream` (SSE stream for multi-message responses). An optional `Mcp-Session-Id` header supports stateful sessions. The server MUST validate the `Origin` header on every incoming connection to prevent DNS rebinding attacks.

## Official SDKs

All SDKs are under the `modelcontextprotocol` GitHub org. Treat version numbers as a snapshot; verify current releases before pinning.

| Language | Package | Install |
|---|---|---|
| TypeScript | `@modelcontextprotocol/sdk` | `npm install @modelcontextprotocol/sdk` |
| Python | `mcp` | `pip install mcp` (or `uv add "mcp[cli]"`) |
| Kotlin | `modelcontextprotocol/kotlin-sdk` | Maven/Gradle — maintained with JetBrains |
| Go | `modelcontextprotocol/go-sdk` | `go get` — maintained with Google |
| Swift | `modelcontextprotocol/swift-sdk` | Swift Package Manager |
| C# | `modelcontextprotocol/csharp-sdk` | NuGet (`ModelContextProtocol`) — maintained with Microsoft; reached v1.0 (WebSearch-corroborated, not independently re-fetched this cycle) |

## Minimal server lifecycle

Regardless of transport, every session follows this sequence:

1. **initialize** — client sends `initialize` with `protocolVersion` and its capabilities; server responds with its `protocolVersion`, `capabilities` (which primitives it supports), and `serverInfo`. The negotiated version governs the session.
2. **initialized** — client sends the `notifications/initialized` notification; the session is now open for normal requests.
3. **discovery** — client may call `tools/list`, `resources/list`, `prompts/list` to enumerate what the server offers.
4. **invocation** — client sends `tools/call` (or equivalent) with the tool name and arguments; server runs the handler and returns a content array.
5. **shutdown** — for stdio, the host closes stdin; for Streamable HTTP, the client sends `HTTP DELETE` with the session ID (if the server issued one).

TypeScript minimal example (using `@modelcontextprotocol/sdk`):

```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const server = new McpServer({ name: "example", version: "1.0.0" });

server.registerTool("ping", {
  description: "Return a pong",
  inputSchema: { message: z.string().optional() },
}, async ({ message }) => ({
  content: [{ type: "text", text: `pong: ${message ?? ""}` }],
}));

const transport = new StdioServerTransport();
await server.connect(transport);
```

## Remote-server concerns

**Authentication** — remote Streamable HTTP servers should implement OAuth 2.1 with PKCE. Full detail is in /resources/mcp-server-authentication.

**Statelessness** — servers that do not issue an `Mcp-Session-Id` are effectively stateless: each POST is independent. This simplifies horizontal scaling but means the server cannot hold in-memory state across requests.

**CORS** — if the server will be called from browser-hosted clients, set `Access-Control-Allow-Origin` appropriately. Restrict origins to known client domains where possible.

**Session handling** — if issuing session IDs, use cryptographically random values (UUID v4 or equivalent). Clients that receive HTTP 404 on a session-bearing request must start a new `initialize` exchange.

## Testing: MCP Inspector

The official interactive testing tool is **MCP Inspector** (`@modelcontextprotocol/inspector` on npm). Run it with:

```bash
npx @modelcontextprotocol/inspector
```

It provides a UI to connect to any MCP server, browse its tools/resources/prompts, send calls manually, and inspect the raw JSON-RPC messages. Source: github.com/modelcontextprotocol/inspector.

## Publishing and discovery

The official registry is at **registry.modelcontextprotocol.io**. To publish, add a `server.json` file to your repository (schema: `https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json`) describing your server's name, version, transport type, and endpoint URL, then use the `mcp-publisher` CLI (`mcp-publisher login github` → `mcp-publisher publish`). Publishing to the registry seeds downstream aggregators and client-vendor marketplaces. For evaluation criteria and security checks to apply before connecting to third-party servers, see /resources/mcp-server-discovery.

## ChangeGamer's /mcp endpoint

ChangeGamer runs a remote MCP server at `https://changegamer.ai/mcp` using the Streamable HTTP transport. It is unauthenticated — no OAuth flow is required to connect. It is open and read-only. Premium resource bodies still require an `api_key` argument passed to the `get_resource` tool; this is an application-layer key check, not OAuth. ChangeGamer does not implement RFC 9728 Protected Resource Metadata.

## Verified sources

- MCP spec (stable 2025-11-25): https://modelcontextprotocol.io/specification/2025-11-25
- MCP spec changelog (2025-11-25 vs 2025-06-18): https://modelcontextprotocol.io/specification/2025-11-25/changelog
- MCP transports spec: https://modelcontextprotocol.io/specification/2025-11-25/basic/transports
- MCP tools concept: https://modelcontextprotocol.io/docs/concepts/tools
- MCP quickstart (server): https://modelcontextprotocol.io/quickstart/server
- TypeScript SDK (npm: @modelcontextprotocol/sdk): https://github.com/modelcontextprotocol/typescript-sdk
- Python SDK (PyPI: mcp): https://github.com/modelcontextprotocol/python-sdk
- C# SDK, reached v1.0 (WebSearch-corroborated via the official repo and Microsoft .NET blog; not independently re-fetched this cycle): https://github.com/modelcontextprotocol/csharp-sdk
- MCP Inspector (npm: @modelcontextprotocol/inspector): https://github.com/modelcontextprotocol/inspector
- Official MCP registry: https://registry.modelcontextprotocol.io/
- MCP registry GitHub (server.json spec): https://github.com/modelcontextprotocol/registry


Streaming Responses for Agents

Transport formats, provider event schemas, and practical concerns for consuming streamed LLM responses in production agents: SSE mechanics, OpenAI (Chat Completions and Responses API) and Anthropic event formats, partial-JSON tool-call parsing, backpressure, cancellation, and gateway proxying.

Streaming lets an agent start processing model output before the full response is complete. For agent builders the three payoffs are: (1) **time-to-first-token (TTFT)** — perceived latency drops because the pipeline can act on early output; (2) **early cancellation** — if the first few tokens reveal a hallucination or wrong tool, the request can be aborted before paying for the full generation; (3) **incremental parsing** — tool-call arguments and structured outputs arrive as partial JSON that can be validated and acted on progressively. See /resources/agent-cost-latency-optimization for the latency framing.

## Transport: Server-Sent Events (SSE)

All three major providers (OpenAI, Anthropic, Google Gemini) stream over HTTP using **Server-Sent Events** (SSE), the W3C/WHATWG standard for unidirectional server-to-client push over a plain HTTP connection. The wire format is `Content-Type: text/event-stream`; each event is one or more `data:` lines terminated by a blank line. Named events use an `event:` field before the `data:` field.

SSE works over HTTP/1.1 (chunked transfer encoding) and HTTP/2 (a single stream). WebSockets are used for **bidirectional** real-time protocols (e.g., OpenAI Realtime API for voice); pure generation streaming uses SSE, not WebSockets.

## OpenAI Chat Completions streaming

Set `"stream": true` in the request body. The response is a sequence of `data:` SSE lines, each carrying a JSON object of type `chat.completion.chunk`. Each chunk has:

- `choices[].delta` — incremental content fragment. On the first chunk, `delta.role` is `"assistant"`. Subsequent chunks carry `delta.content` (text fragment) or `delta.tool_calls` (partial tool-call data).
- `choices[].finish_reason` — `null` during the stream; `"stop"`, `"tool_calls"`, or another terminal value on the final content chunk.

The stream ends with `data: [DONE]` — a sentinel that is not valid JSON and signals the consumer to close the connection.

**Tool calls in OpenAI streaming**: `delta.tool_calls` is a list indexed by position. The first delta for a call includes `id`, `type: "function"`, and `function.name`. Subsequent deltas carry only `function.arguments` as a *partial JSON string fragment*. The consumer must concatenate all `function.arguments` fragments across deltas, then parse the complete string as JSON after `finish_reason: "tool_calls"` is received. See /resources/reliable-tool-calling for schema-validation strategies on the parsed result.

## OpenAI Responses API streaming (newer, recommended for new agent projects)

OpenAI now positions the newer **Responses API** (`/v1/responses`) as the primitive for new agentic projects — Chat Completions stays fully supported, but new capabilities land in Responses first (reported, as of July 2026, per OpenAI's own developer-docs page titles and independent coverage surfaced via WebSearch; direct WebFetch to platform.openai.com and developers.openai.com both returned HTTP 403 this session, so this claim is WebSearch-only, not primary-fetched). Its streaming shape differs structurally from Chat Completions: instead of untyped `chat.completion.chunk` deltas ended by a `[DONE]` sentinel, each SSE event carries a distinct `type` field, closer to Anthropic's scheme — `response.created`, `response.output_item.added`, `response.output_text.delta` (`delta` field for text fragments), `response.output_text.done`, `response.output_item.done`, and `response.completed` (or `response.failed`/`error`). Tool-call arguments stream as `response.function_call_arguments.delta` events keyed by `output_index`, accumulated the same way as Chat Completions' `tool_calls.function.arguments` fragments, then finalized on `response.function_call_arguments.done` or `response.completed`.

## Anthropic Messages streaming

Set `"stream": true` in the request body to `/v1/messages`. Events use both SSE `event:` name fields and a `type` field inside the JSON `data:` payload. The ordered event flow is:

1. `message_start` — contains a `Message` object with empty `content`.
2. For each content block: `content_block_start` → one or more `content_block_delta` events → `content_block_stop`. Each block has an `index` matching its position in the final message.
3. One or more `message_delta` events — top-level message metadata updates (e.g., cumulative `usage` token counts).
4. `message_stop` — stream is complete.

Additional `ping` events may appear anywhere. Error events can arrive mid-stream (e.g., `overloaded_error`); consumers must handle unknown event types gracefully.

**Delta types inside `content_block_delta`:**

- `text_delta` — `delta.text` carries a text fragment.
- `input_json_delta` — `delta.partial_json` carries a partial JSON string fragment for a `tool_use` block's `input` field. Accumulate fragments across deltas and parse the complete string at `content_block_stop`. Current models emit one complete key-value pair per emission, so gaps between events are normal.
- `thinking_delta` — reasoning tokens when extended thinking is enabled.

## Google Gemini streaming

Use `streamGenerateContent` instead of `generateContent`. With the REST API add `?alt=sse` to receive SSE-formatted output. Each SSE `data:` event carries a complete `GenerateContentResponse` JSON object; incremental text arrives in `candidates[0].content.parts[0].text`. There is no separate `[DONE]` sentinel — the stream ends when the HTTP response body closes. Function-call arguments in streaming follow the same accumulate-then-parse pattern as other providers.

## Streaming tool calls and structured outputs

Regardless of provider, function-call arguments arrive as **partial JSON string fragments**. Two handling strategies:

- **Accumulate-then-parse** (simplest): collect all fragments into a buffer; parse the complete JSON string once the block or stream terminates. Safe for all schema shapes.
- **Streaming/partial JSON parser**: libraries such as `partial-json` (npm) or Pydantic's partial JSON parsing mode can deserialize incomplete JSON incrementally, enabling early field access before the stream ends. Useful for long structured outputs where upstream steps can act on early fields.

For validation and schema-enforcement concerns once the full arguments are available, see /resources/reliable-tool-calling.

## Practical concerns for agent builders

**Backpressure and buffering** — if your consumer processes chunks slower than the provider emits them, buffers grow. Size-bound your buffer and apply flow control; for gateway deployments see /resources/ai-gateways-llm-routing.

**Cancellation / abort** — send an HTTP request abort (e.g., `AbortController` in browser or Node.js, `httpx` cancel in Python) to stop generation early. The provider stops decoding; you pay only for tokens generated up to the abort. Ensure your agent loop handles a partial-response state cleanly.

**Error handling mid-stream** — an error event or a dropped TCP connection mid-stream leaves your state machine with a partially assembled response. Track which content blocks received `content_block_stop` (Anthropic) or whether `finish_reason` was set (OpenAI) before treating the response as complete.

**Token accounting** — `usage` fields in streaming responses (OpenAI `stream_options: {"include_usage": true}`; Anthropic `message_delta.usage`) are cumulative, not per-chunk. Read the final value, not a running sum of chunk values.

**Proxying through a gateway** — if you proxy streamed responses through an AI gateway or middleware, ensure the proxy flushes `data:` lines immediately rather than buffering the full response body. A buffering proxy negates all TTFT benefits. See /resources/ai-gateways-llm-routing for gateway selection criteria.

## Verified sources

- WHATWG HTML Living Standard — Server-sent events: https://html.spec.whatwg.org/multipage/server-sent-events.html
- MDN Web Docs — Using server-sent events: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
- Anthropic Messages streaming (event types, input_json_delta, tool use): https://platform.claude.com/docs/en/build-with-claude/streaming
- OpenAI ChatCompletionChunk type (delta fields, tool_calls.function.arguments; re-fetched live 2026-07-18 via raw.githubusercontent.com, confirmed current): https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_chunk.py
- OpenAI Responses API streaming events (WebSearch-fallback only, 4+ agreeing sources — developers.openai.com and platform.openai.com doc-page titles, the OpenAI community forum's "simple guide to events" thread, and vLLM's OpenAI-compatible streaming_events implementation; platform.openai.com/developers.openai.com both 403'd to direct WebFetch this session): https://platform.openai.com/docs/api-reference/responses-streaming/response/output_text/delta
- Google Gemini streaming (streamGenerateContent, GenerateContentResponse, alt=sse): https://ai.google.dev/api/generate-content
- Google Gemini cookbook — Streaming REST quickstart: https://github.com/google-gemini/cookbook/blob/main/quickstarts/rest/Streaming_REST.ipynb


Text-to-SQL and Database Agents

How agents answer questions over structured data by generating and executing SQL: schema context, few-shot prompting, self-correction, safety constraints, benchmarks (Spider, BIRD-SQL), and tooling (LangChain SQLDatabaseToolkit, LlamaIndex NLSQLTableQueryEngine, Vanna, MCP Postgres server).

A text-to-SQL agent accepts a natural-language question, generates a SQL query, executes it against a real database, and synthesizes an answer from the result. The gap between a demo and a production system is wide: schema size, ambiguity, dialect differences, multi-table joins, and query safety all require deliberate engineering.

## Why it is hard

**Schema understanding** — large databases have hundreds of tables and thousands of columns. Feeding the entire schema into the context window is often infeasible and noisy. The model must identify which tables and columns are relevant to the question (schema linking) before writing correct SQL.

**Ambiguity** — natural language is imprecise. "Sales last quarter" requires knowing the calendar convention, time-zone handling, and whether "sales" means orders, invoices, or revenue rows.

**Dialect differences** — SQLite, PostgreSQL, MySQL, BigQuery, Snowflake, and DuckDB differ in date functions, string operators, window syntax, and quoting rules. A prompt tuned for one dialect degrades on another.

**Correctness** — a query can execute without error yet return a wrong answer (wrong JOIN condition, missing filter, off-by-one aggregation). Execution success does not imply semantic correctness.

## Core techniques

**Schema context injection** — provide the model with DDL (CREATE TABLE statements), column descriptions, and 2–3 sample rows per table. This is the single highest-leverage intervention. For large schemas, use retrieval to select only the relevant subset of tables and columns — see /resources/rag-retrieval-for-agents for the retrieval patterns.

**Few-shot examples** — include 3–10 verified question/SQL pairs in the prompt. Examples calibrate the model to your dialect, naming conventions, and common query patterns. Dynamic few-shot (retrieving the most similar examples for each incoming question) consistently outperforms static few-shot on held-out questions.

**Self-correction (run → read error → fix)** — after generating a query, execute it; if it returns a database error, feed the error message back to the model with the original question and schema and ask it to produce a corrected query. One correction round captures the majority of fixable errors. Limit to 2–3 attempts to avoid infinite loops.

**Query validation before execution** — for higher-stakes pipelines, add a validation step: parse the SQL with a library (e.g., `sqlglot`) and check that all referenced table/column names exist in the schema before executing. This catches hallucinated identifiers without a live database round-trip.

**Read-only constraint** — connect to the database with a read-only role (no INSERT/UPDATE/DELETE privileges). This is a hard safety boundary, not a prompt instruction.

## Safety

Text-to-SQL agents combine two injection attack surfaces: the natural-language input (prompt injection) and the generated SQL (SQL injection). A user can craft a question that instructs the model to emit destructive SQL, or stacked statements that escape a read-only transaction. Datadog Security Labs documented a real example: the reference Anthropic Postgres MCP server allowed statement stacking that bypassed its read-only transaction guard (`COMMIT; DROP SCHEMA public CASCADE;` executed successfully), patched by switching to prepared statements.

**Mandatory controls:**

- **Read-only DB role** — enforce at the database connection level, not in the prompt.
- **Row limits** — add `LIMIT N` to every generated query (or enforce via query rewriting) to prevent runaway full-table scans.
- **Allowlist / schema restriction** — expose only the tables the agent needs; deny access to system catalogs and sensitive tables.
- **No statement stacking** — use parameterized queries or prepared statement APIs; do not pass raw model output as a multi-statement string to the database driver.
- **Human approval for writes** — if the use case requires write access, add an explicit human-in-the-loop confirmation step before executing any INSERT/UPDATE/DELETE.

See /resources/agentic-security-checklist (prompt injection mitigations) and /resources/code-execution-sandboxing (database sandboxing patterns).

## Transparency

Return the generated SQL to the caller alongside the result. Agents that hide the SQL make it impossible to audit wrong answers or catch injection attempts. Logged SQL is also the primary signal for fine-tuning and few-shot improvement.

## Benchmarks

**Spider** (Yale LILY Lab) — 10,181 questions over 200 databases across 138 domains, with complex cross-domain SQL. The original benchmark where models now exceed 90% execution accuracy, making it a baseline rather than a frontier test. Official site: https://yale-lily.github.io/spider

**Spider 2.0** (ICLR 2025 Oral) — enterprise-grade successor using real Snowflake, BigQuery, and SQLite databases with large, complex schemas. The best models score roughly 21% success rate, highlighting the gap between toy benchmarks and production workloads. GitHub: https://github.com/xlang-ai/Spider2

**BIRD-SQL** — 12,751 question-SQL pairs over 95 large databases (33.4 GB total) spanning 37 professional domains including dirty/null data and external-knowledge requirements. Evaluates execution accuracy (EX): whether the generated query returns the correct result set. Human performance is ~93% EX; the leaderboard moved fast through mid-2026 — Google's Gemini-SQL2 (built on Gemini 3.1 Pro) posted 80.04% EX in June 2026, and Agentar-Scale-SQL reported 81.67% EX shortly after, both well past the ~72% figure typical earlier in the year. Gains have come mainly from elaborate multi-step pipelines, not raw single-model capability. BIRD-Interact accepted to ICLR 2026 (Oral). Official site: https://bird-bench.github.io/

## Tooling

**LangChain SQLDatabaseToolkit** — a set of tools (`QuerySQLDatabaseTool`, `InfoSQLDatabaseTool`, `ListSQLDatabaseTool`) used with `create_react_agent` (LangGraph). The toolkit handles schema introspection, query execution, and error feedback. Scope credentials to read-only before connecting. Docs: https://python.langchain.com/docs/modules/agents/toolkits/sql_database

**LlamaIndex NLSQLTableQueryEngine** — generates SQL from natural language over a SQLAlchemy- connected database. For schemas too large to fit the context window, `SQLTableRetrieverQueryEngine` dynamically selects relevant tables at query time. Docs: https://developers.llamaindex.ai/python/examples/index_structs/struct_indices/sqlindexdemo/

**Vanna (vanna-ai/vanna)** — open-source Python RAG framework for text-to-SQL. Trains on DDL, documentation strings, and verified question-SQL pairs stored in a vector store; uses retrieved examples as few-shot context at inference time. GitHub: https://github.com/vanna-ai/vanna

**MCP Postgres server** — the reference `@modelcontextprotocol/server-postgres` (now archived at `modelcontextprotocol/servers-archived`) provided read-only PostgreSQL access as an MCP tool. Community successors with configurable access include `crystaldba/postgres-mcp`. All queries should run in READ ONLY transactions; see the Datadog SQL-injection case study for why prepared statements are required even in read-only mode. Archived repo: https://github.com/modelcontextprotocol/servers-archived/tree/main/src/postgres

## Practical checklist

1. Give the model the schema (DDL + column descriptions + sample rows), not just table names.
2. Use dynamic few-shot: retrieve the 3–5 most similar verified examples per question.
3. Connect with a read-only, schema-scoped database role — enforce at the driver, not the prompt.
4. Add a `LIMIT` clause to every generated query.
5. Validate SQL syntax and referenced identifiers before execution.
6. Run the query; on error, feed the error back and retry (max 2–3 rounds).
7. Return the SQL in the response for auditability.
8. For writes, require explicit human approval.

Cross-links: /resources/rag-retrieval-for-agents · /resources/agentic-security-checklist · /resources/code-execution-sandboxing · /resources/evaluating-ai-agents

## Verified sources

- Spider benchmark (Yale LILY Lab): https://yale-lily.github.io/spider
- Spider 2.0 (ICLR 2025 Oral, GitHub): https://github.com/xlang-ai/Spider2
- BIRD-SQL benchmark: https://bird-bench.github.io/
- LangChain SQLDatabaseToolkit docs: https://python.langchain.com/docs/modules/agents/toolkits/sql_database
- LangChain SQL agent API reference: https://reference.langchain.com/python/langchain-community/agent_toolkits/sql/toolkit/SQLDatabaseToolkit
- LlamaIndex text-to-SQL guide: https://developers.llamaindex.ai/python/examples/index_structs/struct_indices/sqlindexdemo/
- Vanna (vanna-ai/vanna) GitHub: https://github.com/vanna-ai/vanna
- MCP Postgres server (archived): https://github.com/modelcontextprotocol/servers-archived/tree/main/src/postgres
- Datadog — SQL injection in the Postgres MCP server: https://securitylabs.datadoghq.com/articles/mcp-vulnerability-case-study-SQL-injection-in-the-postgresql-mcp-server/
- BIRD-SQL leaderboard climb to ~80-82% EX by June 2026 (WebSearch-fallback, 3 sources — MarkTechPost, AI Navigate News, and BIRD leaderboard coverage): https://www.marktechpost.com/2026/06/12/google-releases-gemini-sql2-gemini-3-1-pro-text-to-sql-scores-80-04-on-bird-single-model-leaderboard/


Knowledge Graphs and GraphRAG for Agents

Graph-structured retrieval: when and how to use knowledge graphs over vector RAG for multi-hop, relational, and global corpus queries.

## The problem vector RAG does not solve

Standard vector RAG retrieves chunks by embedding similarity. That works well for
single-hop, fact-lookup questions ("What does X say about Y?") but fails on three
distinct problem classes:

- **Multi-hop / relational** — "How is entity A connected to entity C through B?"
  requires traversing a path that pure cosine similarity cannot reconstruct.
- **Global summarization** — "What are the main themes across this whole corpus?"
  requires aggregating over the entire document set, not retrieving a few chunks.
- **Cross-document entity linking** — the same person, organization, or concept
  appears under different surface forms across documents; a graph merges these into
  one node.

Knowledge graphs model a corpus as **entities** (nodes) and **relationships**
(typed edges), enabling structured traversal at query time.

Cross-link: /resources/rag-retrieval-for-agents for the vector-RAG baseline.

## Core concepts

**Graph construction** — an LLM reads each document chunk and extracts named
entities plus the relationships between them (e.g., `Person –WORKS_AT→ Company`).
This is expensive: each chunk costs one or more LLM calls. The output is a property
graph stored in a graph database.

**Community detection** — clustering algorithms (e.g., Leiden) group densely
connected entities into communities. The Microsoft GraphRAG system
(arxiv:2404.16130) then pre-generates a **hierarchical summary** for each
community using an LLM. These summaries are the backbone of global search.

**Local search** — uses entities as the query entry point. The query is embedded
to find nearest-neighbor entities, then graph traversal expands outward through
relationships and community context to build the prompt. Best for targeted,
entity-specific questions.

**Global search** — instead of traversing, it broadcasts the query across all
pre-computed community summaries, collects partial answers (MAP step), and
aggregates them into a final response (REDUCE step). Best for whole-corpus
sensemaking and thematic questions.

## Approaches and variants

**Microsoft GraphRAG** (`microsoft/graphrag`) — the reference implementation of
the arxiv:2404.16130 paper ("From Local to Global: A Graph RAG Approach to
Query-Focused Summarization"). Produces a full Leiden-community hierarchy with
pre-generated summaries. Also supports a DRIFT search mode that combines global
and local search. Documented at https://microsoft.github.io/graphrag/ and
https://github.com/microsoft/graphrag. Note: not an officially supported Microsoft
product; graph construction is intentionally expensive.

**LightRAG** (`HKUDS/LightRAG`, arXiv:2410.05779, EMNLP 2025) — a lighter
alternative that builds a dual-layer knowledge graph (entities + higher-level
concepts) alongside vector embeddings. Supports five query modes: local, global,
hybrid, naive (pure vector), and mix (default). Incremental graph updates avoid
full re-indexing. GitHub: https://github.com/HKUDS/LightRAG

**HippoRAG** (`OSU-NLP-Group/HippoRAG`, arXiv:2405.14831, NeurIPS 2024) —
inspired by hippocampal indexing theory. Combines knowledge graphs with
Personalized PageRank to model associative memory. Demonstrated up to 20%
improvement on multi-hop QA vs. standard RAG with lower latency than iterative
retrieval approaches. HippoRAG 2 (arXiv:2502.14802, ICML 2025) extends to
continual learning. GitHub: https://github.com/OSU-NLP-Group/HippoRAG

**Hybrid vector + graph** — combine a vector store for chunk retrieval with a
graph store for relational traversal. LlamaIndex `PropertyGraphIndex` and
Neo4j's `neo4j-graphrag-python` package both support this pattern natively.

## Storage

**Property graphs** — nodes and edges carry key-value properties. Neo4j
(Cypher query language) is the dominant choice; others include Amazon Neptune
(supports both property graph and RDF) and Memgraph. Most GraphRAG tooling
targets property graphs.

**RDF / triple stores** — represent facts as subject–predicate–object triples,
queried via SPARQL. Stronger semantic interoperability (W3C standards, ontology
reasoning) but heavier join overhead at scale. Less common in LLM-era agent
pipelines.

**Tooling**:

- **LlamaIndex `PropertyGraphIndex`** — constructs a property graph from
  documents via LLM extraction, stores in a pluggable graph backend
  (Neo4j, in-memory, etc.), and exposes multiple retriever types including
  keyword-entity lookup and vector-based graph node retrieval.
  Docs: https://developers.llamaindex.ai/python/framework/module_guides/indexing/lpg_index_guide/

- **LangChain `langchain-neo4j`** (`GraphCypherQAChain`) — generates Cypher
  queries from natural language against a Neo4j graph. The LLM is given the
  graph schema; it produces a Cypher query, executes it, and reasons over the
  result. Docs: https://python.langchain.com/docs/integrations/graphs/neo4j_cypher/

- **`neo4j-graphrag-python`** — Neo4j's own Python package for building RAG
  pipelines over Neo4j, including a Knowledge Graph Builder pipeline that
  extracts entities from unstructured text.
  Docs: https://neo4j.com/docs/neo4j-graphrag-python/current/

## Tradeoffs

| Dimension | Vector RAG | GraphRAG |
|---|---|---|
| Build cost | Low (embed chunks) | High (many LLM calls per chunk) |
| Update / freshness | Re-embed changed chunks | Re-extract affected subgraph |
| Multi-hop queries | Poor | Strong |
| Global summarization | Poor | Strong |
| Operational complexity | Low | High |
| Best for | Fact lookup, semantic search | Relational, entity-centric, whole-corpus |

Use GraphRAG when: your queries span multiple entities and require traversing
relationships; you need "what is the overall picture" summaries; or entities
appear under many surface forms across documents.

Stick with plain RAG when: questions are single-hop or semantic; the corpus
is small or fast-changing; or build cost/latency constraints are tight.

## Agentic angle

An agent can treat a knowledge graph as a **tool**: issue graph queries
(Cypher, SPARQL, or a higher-level API) as discrete tool calls, inspect the
subgraph returned, and decide whether to traverse further. This fits naturally
into MCP or function-calling patterns — each traversal step is a tool call
with a verifiable intermediate result.

The most robust production pattern combines vector retrieval (fast chunk
lookup) with graph traversal (relational context): the vector index answers
"what chunks are relevant?" and the graph answers "how are these entities
connected?".

Cross-links: /resources/reliable-tool-calling · /resources/embeddings-vector-search · /resources/agent-memory-context

## Verified sources

- Microsoft GraphRAG repo: https://github.com/microsoft/graphrag
- Microsoft GraphRAG docs: https://microsoft.github.io/graphrag/
- GraphRAG paper (arXiv:2404.16130): https://arxiv.org/abs/2404.16130
- LightRAG repo (HKUDS, arXiv:2410.05779): https://github.com/HKUDS/LightRAG
- HippoRAG repo (OSU-NLP-Group, arXiv:2405.14831): https://github.com/OSU-NLP-Group/HippoRAG
- HippoRAG 2 (arXiv:2502.14802): https://arxiv.org/abs/2502.14802
- LlamaIndex PropertyGraphIndex guide: https://developers.llamaindex.ai/python/framework/module_guides/indexing/lpg_index_guide/
- LangChain Neo4j integration: https://python.langchain.com/docs/integrations/graphs/neo4j_cypher/
- Neo4j GraphRAG Python package: https://neo4j.com/docs/neo4j-graphrag-python/current/


Testing AI Agents in CI

How to write deterministic, fast, CI-friendly tests for non-deterministic agents: the three-layer test pyramid, LLM mocking, cassette/VCR-style replay, snapshot testing of tool-call trajectories, pass@k thresholds, and verified tooling.

The core tension: agents are non-deterministic, but CI pipelines need tests that are deterministic, fast, cheap, and reliable. You can get both — by being deliberate about which layer of the stack you test at which level.

## The three-layer test pyramid for agents

### Layer 1 — deterministic unit tests (run on every commit)

Test the code *around* the model: tool functions, parsers, prompt-template renderers, schema validators, retry logic, and output-format coercers. Mock or stub the LLM client entirely. These tests are ordinary unit tests — no API calls, no network, fast and free. They catch the majority of regressions because most bugs live in the glue, not the model.

### Layer 2 — recorded/replayed LLM interactions (run on every commit)

Use cassette/VCR-style fixtures: the first time a test runs, it hits the real API and serialises the full HTTP exchange to a YAML file. Every subsequent run replays that cassette instead of making a live call — fast, free, and network-independent. Commit cassettes to version control. Re-record only when prompt templates or schema change. In CI, pass `--record-mode=none` (pytest-recording's actual flag — not `--vcr-record`, which belongs to the separate, unrelated `pytest-vcr` package) so a missing cassette is a test failure, not a live API call.

**Important**: scrub credentials and sensitive headers from cassettes before committing. VCR.py and pytest-recording both support `filter_headers` and `filter_post_data_parameters` for this.

### Layer 3 — live smoke / eval tests (nightly or pre-release, NOT per-commit)

A small, hand-curated set of end-to-end tasks run against the real model. Gate these on a separate CI job (nightly, or triggered manually for releases). They are expensive and inherently flaky — keep the set small and treat failures as signals, not hard blockers per PR.

## Key techniques

**Mocking the LLM client** — use `unittest.mock.patch` or a dependency-injection seam to replace the model client with a fixture that returns a pre-canned structured response. This is the cheapest form of Layer 1 testing.

**temperature=0 and seeds** — setting temperature to zero and a fixed seed reduces variance but does not guarantee bit-for-bit identical outputs across runs. Floating-point non-associativity from GPU batching and MoE routing means the same prompt can yield different tokens in different batch contexts. Never rely on temperature=0 as a substitute for proper mocking or cassette replay.

**Snapshot testing of tool-call sequences** — store the expected sequence of tool calls (names + arguments) as a snapshot. Assert on structure and argument values, not on the free-text reasoning. A tool-call diff in CI surfaces unintended trajectory changes before they reach production. Cross-link: /resources/reliable-tool-calling.

**Structured-output assertions** — if your agent emits JSON or a Pydantic schema, assert against the schema and the key field values, not against the exact prose. This tolerates benign rephrasing while catching real regressions.

**LLM-as-judge in tests** — a second model grades the output against a rubric. Useful for Layer 3 smoke tests but carries its own flakiness: the judge model can disagree with itself across runs. Treat judge scores as soft signals; set wide pass/fail thresholds and aggregate over multiple runs. See /resources/evaluating-ai-agents for benchmark-grade eval methodology.

## Handling flakiness

- **Separate suites** — keep deterministic (Layers 1–2) and probabilistic (Layer 3) suites in distinct test files and CI jobs. Never let a non-deterministic test gate a PR.
- **pass@k thresholds** — for probabilistic tests, run k trials and assert that at least m succeed (e.g., pass@5 with m=4). This is more honest than a single run and absorbs natural variance without hiding real regressions.
- **Retries vs quarantine** — automatic retries mask real failures; prefer quarantining a flaky test into the nightly suite until its failure mode is understood.
- **Cost controls** — set token-budget limits per test job. Tag each Layer 3 job with expected cost and alert when actual cost drifts more than 20%. Cross-links: /resources/agent-cost-latency-optimization, /resources/agent-observability (traces from production runs can seed new cassettes and test cases).

## Verified tooling

**VCR.py** (`vcrpy`) — Python HTTP record/replay. Intercepts HTTP at the library level; serialises to YAML cassettes. Works with any HTTP-based LLM SDK.

**pytest-recording** — a pytest plugin wrapping VCR.py. Adds `--record-mode` CLI option and `@pytest.mark.vcr` decorator. Maintained by kiwicom on GitHub.

**promptfoo** — YAML-driven test runner for prompts and agents with native CI/CD integration (GitHub Action available). Supports structured assertions (`is-json`, `contains-json`, `llm-rubric`), cost/latency thresholds, and red-teaming. MIT-licensed; acquired by OpenAI in March 2026 but remains open-source.

**DeepEval** — pytest-based LLM evaluation framework. `assert_test()` and `deepeval test run` plug directly into CI pipelines; supports parallel execution via `-n` flag. Maintained by Confident AI.

## Verified sources

- VCR.py repo (kevin1024/vcrpy): https://github.com/kevin1024/vcrpy
- VCR.py docs: https://vcrpy.readthedocs.io/en/latest/
- pytest-recording repo (kiwicom): https://github.com/kiwicom/pytest-recording
- pytest-recording on PyPI: https://pypi.org/project/pytest-recording/
- promptfoo CI/CD integration docs: https://www.promptfoo.dev/docs/integrations/ci-cd/
- promptfoo GitHub: https://github.com/promptfoo/promptfoo
- DeepEval unit testing in CI/CD: https://deepeval.com/docs/evaluation-unit-testing-in-ci-cd
- DeepEval GitHub (confident-ai): https://github.com/confident-ai/deepeval
- temperature=0 non-determinism explained: https://www.zansara.dev/posts/2026-03-24-temp-0-llm/


Generative UI and Agent-to-UI Protocols

How agents drive UI dynamically: the AG-UI protocol, framework options (Vercel AI SDK, CopilotKit, assistant-ui, LangGraph), streaming component patterns, and human-in-the-loop UI design.

## What generative UI means for agents

Classical agents return text. Generative UI agents render or drive interface components dynamically at runtime — streaming partial UI, triggering tool-call-driven component swaps, surfacing agent state visually, and pausing for human approval before continuing. The result is a UI whose structure is determined at inference time, not hardcoded at build time.

This is architecturally distinct from chat streaming (plain text deltas). An agent's output is a mixed stream of text chunks, tool calls, state snapshots, and lifecycle signals. The frontend needs a standard way to receive and render all of those incrementally. For the underlying streaming mechanics see /resources/streaming-for-agents.

## The agent↔UI protocol problem

Without a protocol, every team builds a bespoke event contract between their agent backend and their frontend. This leads to tight coupling: the UI breaks when the agent framework changes, and any new agent framework requires a new frontend adapter.

A dedicated agent↔UI event protocol solves this by defining a typed event vocabulary — the same way MCP defines agent↔tool communication and A2A defines agent↔agent communication. For the full protocol-layering picture see /resources/mcp-vs-a2a.

## AG-UI: the Agent-User Interaction Protocol

AG-UI (github.com/ag-ui-protocol/ag-ui) is an open, lightweight, event-based protocol that standardizes how AI agents connect to user-facing applications. It is maintained by the ag-ui-protocol organization on GitHub and originated from CopilotKit, which remains the primary steward. As of July 2026 the repo has 14.8k+ stars and 32+ releases (latest dated July 15, 2026; confirmed via a live GitHub fetch this cycle). The repo's own README now lists adopters across four tiers: 1st-party support (Microsoft Agent Framework, Google ADK, AWS Strands Agents, Mastra, Pydantic AI, Agno, LlamaIndex, AG2), partnership integrations (LangGraph, CrewAI), community support (Claude Agent SDK, Langroid), and infrastructure/spec adopters (Amazon Bedrock AgentCore, Oracle Agent Spec).

**How it works.** The frontend sends a single HTTP POST to the agent endpoint and then listens to a Server-Sent Events (SSE) stream. The agent emits typed events describing its actions; the frontend reacts to them. The protocol is transport-agnostic (SSE, WebSocket, or WebHook); SSE is the canonical transport.

**Verified event categories and types (16 total):**

- *Lifecycle:* `RUN_STARTED`, `RUN_FINISHED`, `RUN_ERROR`
- *Text message:* `TEXT_MESSAGE_START`, `TEXT_MESSAGE_CONTENT`, `TEXT_MESSAGE_END`
- *Tool calls:* `TOOL_CALL_START`, `TOOL_CALL_ARGS`, `TOOL_CALL_END`, `TOOL_CALL_RESULT`
- *State management:* `STATE_SNAPSHOT`, `STATE_DELTA`
- *Special:* `INTERRUPT`, `CUSTOM`

The `STATE_DELTA` event enables bi-directional state sync: both the agent backend and UI components can read from and write to a shared state layer in real time. `INTERRUPT` is the HITL mechanism — the agent pauses, emits an interrupt payload, and waits for the UI to send a resume command.

AG-UI complements rather than replaces MCP (agent↔tools) and A2A (agent↔agent): the three protocols address different edges of the agent graph.

## Framework approaches (verified)

**Vercel AI SDK — `streamUI` / AI SDK RSC** (ai-sdk.dev). The `streamUI` function streams React Server Components alongside LLM output: tool calls can return React components directly, enabling tool-call-as-UI. Status as of 2026: AI SDK RSC is experimental and development is paused; Vercel recommends `useChat` (AI SDK UI, client-side hooks) for production. A newer json-render pattern replaces direct RSC streaming with schema-validated JSON that the client maps to registered components.

**CopilotKit** (docs.copilotkit.ai). The primary maintainer of AG-UI. Provides React hooks and components for generative UI, shared state, and HITL flows on top of the AG-UI event stream. Supports React, Angular, Vue, React Native, and Slack. Includes `useCopilotAction` for rendering interactive approval components mid-flight.

**assistant-ui** (assistant-ui.com). Open-source TypeScript/React component library (Radix-UI-inspired composable primitives) for building agent chat UIs. First-class integrations with LangGraph and Vercel AI SDK. Handles streaming, auto-scroll, retries, tool-call rendering, markdown, and accessibility out of the box.

**LangGraph streaming + interrupt model** (docs.langchain.com). LangGraph emits state deltas via `stream_mode="updates"` (preferred; bandwidth-efficient) or full state via `stream_mode="values"`. Interrupt nodes pause graph execution at predefined checkpoints; the `useStream` hook in compatible frontends surfaces `stream.interrupt` for rendering approval UI. Persistence is handled via checkpointers for durable HITL.

## Key patterns and concerns

**Streaming partial UI.** Render a skeleton/loading component immediately when a tool call starts; swap in the real component when the tool result arrives. Avoids blank states during long agent operations.

**Tool-call-as-UI.** An agent tool call *is* a UI action: `show_chart(data)` triggers a chart component, not a text description. Tool schemas define the contract between agent and UI.

**Human-in-the-loop approval components.** HITL pauses are a first-class pattern: the agent emits an `INTERRUPT` (AG-UI) or a checkpointed interrupt node (LangGraph), the UI renders an approve/reject/edit form, and execution resumes on user confirmation. See /resources/multi-agent-orchestration-patterns for HITL architecture.

**State synchronization.** AG-UI's `STATE_SNAPSHOT` / `STATE_DELTA` events enable the frontend to track agent-side state without polling. Both sides can write to the shared state, making collaborative UIs (agent + human editing the same document) tractable.

**Agent-driven UI as an elevated risk surface.** An agent that can render arbitrary components or execute UI actions on behalf of the user is a higher-privilege principal than a chatbot. Validate all tool-call inputs against a strict schema before rendering; treat agent-generated UI content as untrusted data, not trusted code. See /resources/agentic-security-checklist.

## Verified sources

- AG-UI protocol repo (ag-ui-protocol/ag-ui) — star/release counts and adopter tiers re-fetched live 2026-07-18 via GitHub and the repo's raw README: https://github.com/ag-ui-protocol/ag-ui
- AG-UI 17 event types (CopilotKit blog; note the live repo's own README currently says "~16 standard event types," matching this entry's body — the blog title's "17" is left uncorrected pending its own re-check): https://www.copilotkit.ai/blog/master-the-17-ag-ui-event-types-for-building-agents-the-right-way
- AG-UI on Amazon Bedrock AgentCore: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-agui-protocol-contract.html
- CopilotKit repo (GitHub): https://github.com/CopilotKit/CopilotKit
- CopilotKit generative UI docs: https://www.copilotkit.ai/generative-ui
- Vercel AI SDK RSC — streamUI reference: https://ai-sdk.dev/docs/reference/ai-sdk-rsc/stream-ui
- Vercel AI SDK — Streaming React Components: https://ai-sdk.dev/docs/ai-sdk-rsc/streaming-react-components
- Vercel AI SDK 3.0 generative UI announcement: https://vercel.com/blog/ai-sdk-3-generative-ui
- assistant-ui repo (assistant-ui/assistant-ui): https://github.com/assistant-ui/assistant-ui
- LangGraph interrupt + HITL: https://docs.langchain.com/oss/python/langchain/frontend/human-in-the-loop


Fine-Tuning vs RAG vs Prompting

Decision guide for agent builders: when to use prompting, RAG, or fine-tuning — and how they combine. Covers SFT, LoRA/QLoRA, DPO, distillation, and a symptom-to-fix table.

## Key facts

- Prompting, RAG, and fine-tuning each solve a different kind of problem, and production systems typically layer more than one together rather than picking a single winner.
- Adjusting your prompt is the cheapest lever available — it costs nothing to try and can be undone instantly, so it should be the first thing you attempt.
- RAG works by pulling in outside content at query time so the model can draw on information it was not trained with — the right call whenever facts are proprietary, change frequently, or simply will not fit in the weights.
- Fine-tuning reshapes how a model behaves — its tone, output formatting, or a narrow skill — but it is a poor way to inject new information, since anything learned during training gradually falls out of date.
- LoRA/QLoRA are parameter-efficient fine-tuning methods that freeze base weights and train small low-rank adapters, cutting GPU memory and multi-task serving cost.
- Work through the techniques in sequence — better prompts first, then retrieval if the gap is missing or outdated knowledge, and save fine-tuning for stubborn behavioral issues neither of the other two can resolve.

## The mental model

These three techniques solve different problems and are routinely combined in production systems. Treating them as competitors leads to the wrong choice every time.

- **Prompting** — changes what you *ask* the model, at zero cost. The right first lever.
- **RAG** — changes what the model *knows* at inference time by injecting external content into context. Use when the knowledge is proprietary, current, or too large for the weights.
- **Fine-tuning** — changes how the model *behaves*: its style, format reliability, or a narrow skill. Does not reliably teach fresh facts (weights go stale; fine-tuned knowledge does not update itself).

Key rule: **knowledge that changes often belongs in RAG, not fine-tuned weights.**

## Try in this order

1. **Prompting first** — clearer instructions, few-shot examples, structured delimiters, output schemas. See /resources/prompt-context-engineering. Fastest to iterate; fully reversible.
2. **Add RAG** before fine-tuning if the bottleneck is missing or stale knowledge. See /resources/rag-retrieval-for-agents.
3. **Fine-tune** only when you have a persistent behavioral defect that prompting and RAG cannot fix, and you have enough high-quality labeled examples to train on.

## Fine-tuning methods

**Supervised fine-tuning (SFT)** — train on input/output pairs demonstrating desired behavior. OpenAI documents SFT as a supported method for style, format, and task adaptation (platform.openai.com/docs/guides/supervised-fine-tuning).

**Parameter-efficient fine-tuning (PEFT)** — instead of updating all weights, inject small trainable matrices. The dominant method is LoRA (Low-Rank Adaptation, Hu et al., arXiv:2106.09685): freeze the base weights and add a pair of low-rank matrices (W = W₀ + AB) to each transformer layer. QLoRA extends LoRA by quantizing the base weights to 4-bit before adding the adapters, dramatically reducing GPU memory requirements. PEFT methods produce swappable adapter files that share the base model, making multi-task serving much cheaper than keeping separate full copies.

**Preference fine-tuning (DPO / RLHF)** — align the model to human preferences via ranked pairs of outputs (preferred vs. rejected). RLHF (Reinforcement Learning from Human Feedback) uses a learned reward model and policy-gradient updates. DPO (Direct Preference Optimization, Rafailov et al., arXiv:2305.18290) simplifies this: it directly optimizes a classification-style loss over preference pairs, eliminating the separate reward model and RL training loop, while matching or exceeding RLHF quality. Standard practice is SFT first, then DPO.

**Distillation** — train a smaller model to mimic a larger one's outputs on a narrow task. Use when you need a smaller, cheaper, faster model that matches a frontier model on a specific task. Requires a dataset of (input, large-model-output) pairs. Cross-link: /resources/open-weight-models-for-agents for which base models are fine-tunable.

## Symptom-to-fix table

| Symptom | Likely fix |
|---|---|
| Model lacks current or proprietary facts | RAG |
| Output format or schema is unreliable | Better prompt + structured outputs; fine-tune (SFT) if persistent |
| Tone or style is wrong | Improve system prompt; fine-tune (SFT) if consistent across inputs |
| Model too slow or expensive at scale | Distillation/fine-tune a smaller model; see /resources/agent-cost-latency-optimization |
| Model makes tool-calling mistakes | Structured output + typed schemas; SFT on tool-call examples |
| Need model to follow complex instructions reliably | Few-shot prompting first; SFT if it fails at scale |
| Behavior must reflect human ranking preferences | DPO or RLHF after SFT |

## Honest tradeoffs of fine-tuning

Fine-tuning carries real costs that teams underestimate:

- **Data preparation** — collecting, cleaning, and labeling high-quality training examples is the hardest part. Diverse, high-quality data matters more than quantity.
- **Training and evaluation infrastructure** — requires GPU compute, experiment tracking, and offline evaluation before each deploy.
- **Serving a custom model** — you now own the hosting of a bespoke artifact. When the base model is updated by the provider, your fine-tuned version stays behind.
- **Staleness** — fine-tuned knowledge does not update itself. Mixing RAG into the fine-tuned model's inference pipeline is the standard production pattern to keep knowledge current.
- **Provider platform risk** — OpenAI has been winding down self-serve fine-tuning through 2026: organizations with no prior fine-tuning history lost the ability to start new jobs in May 2026, restrictions tightened further in July 2026, and a full stop on creating new fine-tuning jobs for all customers is slated for January 2027 (already-deployed fine-tuned models keep serving inference until their base model is retired). OpenAI's stated reason is that newer base models plus prompting and RAG now cover most cases fine-tuning used to fix — reinforcing this guide's "try prompting and RAG first" ordering. This claim is WebSearch-only, not primary-fetched (platform.openai.com and the OpenAI community/blog posts describing it returned HTTP 403 to direct fetch this session) — verify current status on platform.openai.com/docs/guides/fine-tuning before committing to an OpenAI-hosted fine-tuning pipeline.

## Combining all three

The 2026 production default for complex agents is: a fine-tuned (or instruction-tuned) model that has been preference-aligned, served with RAG for current knowledge, and steered per-request via structured system prompts. These layers are additive: adding RAG to a fine-tuned model is normal; adding a better system prompt to a RAG-augmented fine-tuned model is normal.

Cross-links: /resources/rag-retrieval-for-agents · /resources/prompt-context-engineering · /resources/reliable-tool-calling · /resources/agent-cost-latency-optimization

## Verified sources

- LoRA paper (Hu et al., 2021): https://arxiv.org/abs/2106.09685
- DPO paper (Rafailov et al., 2023): https://arxiv.org/abs/2305.18290
- OpenAI model optimization guide: https://platform.openai.com/docs/guides/fine-tuning
- OpenAI supervised fine-tuning guide: https://platform.openai.com/docs/guides/supervised-fine-tuning
- OpenAI DPO guide: https://platform.openai.com/docs/guides/direct-preference-optimization
- OpenAI fine-tuning best practices: https://platform.openai.com/docs/guides/fine-tuning-best-practices


Synthetic Data Generation for Agent Training

How to build agentic training corpora without human annotation at scale: the three generation patterns (distillation, self-play, environment rollout), open pipelines (distilabel, AgentInstruct, APIGen-MT, TOUCAN), quality filtering, and the model-collapse risk.

Frontier agents need training data that captures multi-step tool use, error recovery, and multi-turn reasoning — patterns that are expensive and slow to collect from humans at scale. Synthetic data generation fills this gap, but introduces its own failure modes. This guide covers the three principal patterns, the production-ready open pipelines, the quality controls that separate useful data from noise, and the collapse risk that shapes every design decision.

## The three generation patterns

**1. Teacher distillation.** A stronger ("teacher") model generates instruction–response pairs used to post-train a smaller ("student") model. The student inherits the teacher's behavior at a fraction of the inference cost. Quality depends on the teacher's accuracy and the diversity of prompts. Distillation is the dominant method for instruction-following (SFT) data; frontier labs treat their human preference data as a moat precisely because distillation cannot fully replicate it.

**2. Self-play / iterative refinement.** The model generates a candidate, critiques it against a rubric or constitution, and refines it until a verifier (a reward model or rule-based checker) accepts the output. Anthropic's Constitutional AI is the earliest documented large-scale application. Reinforcement Learning from Verifiable Rewards (RLVR) extends this to tasks with ground-truth answers (math, code, tool-call correctness), where the verifier is deterministic rather than model-based.

**3. Environment rollout.** An agent executes actions in a real or simulated environment (a browser, a code interpreter, a set of MCP servers), and the resulting (observation, action, outcome) trajectories become training data. This produces the most agentic data but requires a live environment or a high-fidelity simulator, and quality depends heavily on the rollout policy and task diversity.

## Production pipelines

| Pipeline | Pattern | Notable | License |
|---|---|---|---|
| distilabel (Argilla) | Distillation, preference, critique | Unified pipeline over many providers; Ray-distributed | Apache 2.0 |
| AgentInstruct (Microsoft) | Multi-agent flows from raw documents | 25M pairs (1M public subset) | dataset CDLA-Permissive-2.0 |
| APIGen-MT (Salesforce) | Two-phase: task blueprints + simulated agent–human trajectories | Multi-turn tool use; xLAM-2 models open-sourced | open (data + models) |
| TOUCAN (IBM / UW) | Environment rollout from ~500 real MCP servers | 1.5M trajectories, 2,000+ tools | Apache 2.0 dataset |

**distilabel** (github.com/argilla-io/distilabel) is the most general-purpose open framework. It provides a unified pipeline abstraction over dozens of LLM providers, implements techniques from published papers (instruction following, dialogue generation, preference modeling), and supports distributed execution via Ray. Maintenance note (as of July 2026): the project's own README states the original Argilla authors have moved to other projects and a group of community collaborators has taken over maintenance and is preparing the next release — check current release cadence before depending on it for a production pipeline.

**AgentInstruct** generates both prompts *and* responses from raw seed documents using a multi-agent flow with reflection, avoiding the need for a pre-existing prompt set. A Mistral-7B post-trained on the full 25M-pair corpus improved 40% on AGIEval, 54% on GSM8K, and 45% on AlpacaEval over the untuned base (Microsoft Research, 2024). The 1M-pair public subset is at microsoft/orca-agentinstruct-1M-v1 on Hugging Face.

**APIGen-MT** (arXiv:2504.03601) targets multi-turn agentic tool use. Phase 1 produces task blueprints with ground-truth action sequences via an LLM-committee review; phase 2 instantiates them as full trajectories via simulated human–agent interplay. The resulting xLAM-2-70b-fc-r scores 56.2% on τ-bench and outperforms GPT-4o and Claude 3.5 on τ-bench and BFCL — and smaller xLAM-2 models often beat larger ones in multi-turn settings. 5K sample trajectories and the xLAM-2-fc-r models (1B–70B) are open-sourced.

**TOUCAN** (arXiv:2510.01179, IBM Research / Univ. of Washington) is the largest public tool-agentic dataset: 1.5M trajectories synthesized from ~500 real-world MCP server environments spanning 2,000+ tools. Five models generate task queries; three teacher models and two agentic frameworks construct trajectories; model-based filtering removes low-quality samples. Dataset: Agent-Ark/Toucan-1.5M (Hugging Face).

## Quality filtering is not optional

Raw synthetic output degrades rapidly without filtering. The main failure modes:

- **Hallucinated tool calls** — a teacher invents tool names or argument schemas absent from the real API. Filter: execute the call against the real environment and reject any trajectory that produces a tool-not-found or schema-validation error.
- **Reward hacking** — self-play against a learned reward model can produce outputs that score well on the proxy but fail on held-out real tasks. Mitigation: prefer rule-based verifiers (unit tests, execution outcome, format checks) over model judges where the task allows.
- **Low diversity / mode collapse** — distillation from a single teacher with homogeneous prompts overrepresents the teacher's most probable outputs. Mitigation: seed from diverse real sources, deduplicate at the embedding level, and measure n-gram entropy of the corpus.

## The model-collapse risk

Training successive model generations on an increasingly synthetic corpus — without retaining real human data — produces model collapse: progressive loss of rare, nuanced, and long-tail patterns that amplifies errors from prior generations. The research consensus practitioners act on:

- **Never fully replace human data with synthetic data.** Keep a substantial real-data share; treat the synthetic proportion as a ceiling, not a floor.
- **Accumulate, do not substitute.** Keeping the original human signal alongside synthetic augmentation prevents the recursive degradation loop.
- **Track provenance.** A growing share of scraped web text is itself AI-generated, so "human data" must be verified, not assumed.
- **Use grounding signals.** Data anchored to execution outcomes (code runs, tool calls succeed/fail, math answers check out) is far more collapse-resistant than purely generative text.

## Related ChangeGamer resources

- When to fine-tune versus prompt or retrieve: /resources/fine-tuning-vs-rag
- Evaluation methodology for the resulting models: /resources/evaluating-ai-agents
- Testing fine-tuned agents in CI: /resources/testing-ai-agents
- RAG as a complement to fine-tuning for knowledge tasks: /resources/rag-retrieval-for-agents

## Verified sources

- distilabel — Argilla synthetic-data framework (Apache 2.0), README confirms community-led maintenance transition as of July 2026: https://github.com/argilla-io/distilabel
- AgentInstruct — Microsoft Research blog: https://www.microsoft.com/en-us/research/blog/orca-agentinstruct-agentic-flows-can-be-effective-synthetic-data-generators/
- AgentInstruct paper (arXiv:2407.03502): https://arxiv.org/abs/2407.03502
- orca-agentinstruct-1M-v1 dataset (Hugging Face): https://huggingface.co/datasets/microsoft/orca-agentinstruct-1M-v1
- APIGen-MT paper (arXiv:2504.03601): https://arxiv.org/abs/2504.03601
- APIGen-MT project page: https://apigen-mt.github.io/
- TOUCAN paper (arXiv:2510.01179): https://arxiv.org/abs/2510.01179
- TOUCAN dataset — Agent-Ark/Toucan-1.5M (Hugging Face): https://huggingface.co/datasets/Agent-Ark/Toucan-1.5M
- TOUCAN code: https://github.com/TheAgentArk/Toucan
- RLHF Book — Synthetic Data chapter (Nathan Lambert): https://rlhfbook.com/c/12-synthetic-data


Data Privacy and PII for Agents

How autonomous agents expose PII — context ingestion, tool calls, memory, logs — and the controls that contain it: detection, redaction, data minimization, provider ZDR tiers, GDPR, EU AI Act, CCPA, and a practical compliance checklist.

**Not legal advice.** This guide is informational. Consult qualified legal counsel for compliance decisions specific to your organization.

## Key facts

- Agents expand PII exposure along four dimensions a bounded single API call doesn't have: large ingested context, external tool calls that can transfer data to third parties, persistent memory/logs, and the risk that a provider trains on request data.
- Core controls are catching and masking sensitive fields ahead of the model call or any write to storage, sending a task only the specific data it needs rather than a whole record, and scrubbing observability exports at the point they leave the system.
- Retrieved and tool-output content should be treated as potentially sensitive too, since a RAG excerpt can itself carry names or financial/health data — not just what the user typed.
- Major providers differ in default retention: OpenAI keeps standard API logs up to 30 days before deletion (with Zero Data Retention available on approval for eligible endpoints), while Anthropic cut its standard retention from 30 to 7 days as of September 2025.
- GDPR's right to erasure is hard to reconcile with how agents actually store data, because information a model has already absorbed into its parameters, or stored inside a long-lived embedding index, generally can't be surgically deleted short of a retrain or an index rebuild — a gap regulators flagged as an enforcement priority in 2025.
- The EU AI Act's high-risk Annex III obligations, originally due August 2026, have since been finalized as deferred to December 2, 2027, following European Parliament and Council approval in June 2026 (with a separate, later deferral for Annex I embedded-AI products).
- CCPA/CPRA deletion rights are understood to extend to AI training data, which regulators have signaled may require retraining large foundation models on request.

## Why agents are a distinct privacy risk

A standard API call has a bounded blast radius: you send a request, you get a response. Agents are different along four dimensions that dramatically expand PII exposure:

1. **Large ingested context.** Agents receive and reason over full documents, email threads, CRM records, and tool outputs — all of which may contain PII. The model "sees" everything in its context window.
2. **External tool calls.** Every tool call that leaves your infrastructure is a potential data transfer. A retrieval tool, a search API, or a calendar integration can pass PII to a third-party endpoint outside your control.
3. **Memory and persistent logs.** Agents that write to memory stores or emit detailed observability traces may persist PII long after the task ends. LLM provider logs, LangSmith traces, and application databases are all surfaces.
4. **Training risk.** Unless your contract explicitly prohibits it, some providers may use request/response data to improve their models. PII ingested today could influence model outputs tomorrow.

Map the PII flow for your agent: user input → prompt assembly → model inference → tool call → model response → memory write → log line. Each arrow is a potential leak.

## Core controls

**PII detection and redaction before send.** Run a PII detector on all content before it enters the model context or is written to any external store. Replace detected entities with tokens or placeholders (`<PERSON_0>`, `<EMAIL_0>`) and pass only the redacted form downstream. Restore originals only where the task requires it, in a controlled environment.

**Data minimization.** Send only the fields and records that the current task actually requires. If the agent needs an order status, pass the order ID and status — not the full customer record. Data that never enters the context window cannot leak from it.

**No raw PII in logs or traces.** Redact PII before writing observability data. Structured log fields like `user_id` are acceptable; raw prompt strings containing names, SSNs, or health data are not. See /resources/agent-observability for the broader tracing pattern; apply redaction at the exporter layer so all downstream sinks (LangSmith, Datadog, S3) receive only sanitized data.

**Short retention windows.** Define explicit retention limits for every store that touches agent data: prompt logs, memory databases, vector indices, and audit trails. Data that is deleted cannot be breached.

**Treat retrieved and tool-output data as potentially sensitive.** A RAG retrieval returning a contract excerpt may contain names, financial figures, or health data. Apply the same redaction and minimization rules to content the agent reads as to content the user sends.

Cross-links: /resources/agent-observability (redact traces) · /resources/agentic-security-checklist (agent security posture).

## Provider data handling

Major API providers offer enterprise data terms that address training and retention.

**OpenAI API.** By default, OpenAI does not use API inputs or outputs to train models. Standard API logs are retained for up to 30 days for abuse detection, then deleted. Enterprise customers with a qualifying use-case can request Zero Data Retention (ZDR) for eligible endpoints, which eliminates the 30-day abuse-monitoring log. ZDR requires prior approval. Source: openai.com/enterprise-privacy/

**Anthropic API.** Anthropic does not train models on commercial API inputs or outputs by default. As of September 2025 the standard API log retention was reduced from 30 days to 7 days. Enterprise/business accounts can negotiate Zero Data Retention terms via the Data Processing Addendum. Source: privacy.anthropic.com

Always execute a signed Data Processing Addendum (DPA) with any provider that processes personal data on your behalf — this is a mandatory requirement under GDPR Article 28. For regulated workloads (health, finance, legal), verify that the specific endpoint and model you use are covered by ZDR terms before sending any PII.

## Regulation (high level)

**GDPR (Regulation (EU) 2016/679).** The primary framework for EU-resident personal data. Key obligations for agent builders: establish a lawful basis for each processing activity (Article 6); apply data minimization — collect only what is necessary (Article 5(1)(c)); respect the right to erasure (Article 17, "right to be forgotten"); execute a DPA with every sub-processor (Article 28); apply standard contractual clauses or equivalent safeguards for data transferred outside the EEA.

The "right to be forgotten" creates a structural tension for agents: personal data baked into model weights or a persistent vector memory is technically difficult to erase without retraining or rebuilding the index. This is the machine-unlearning problem. The EDPB made the right to erasure its 2025 coordinated enforcement priority; regulators expect organizations to have a documented strategy for handling deletion requests even when full erasure from weights is infeasible.

**EU AI Act (Regulation (EU) 2024/1689).** A risk-tiered framework for AI systems. Four tiers: (1) prohibited practices (e.g., social scoring by public authorities); (2) high-risk systems (Annex III — biometrics, employment screening, credit scoring, critical infrastructure), subject to documentation, logging, human oversight, and conformity obligations; (3) limited-risk systems (chatbots, synthetic media), subject to transparency obligations (users must know they are interacting with AI — transparency rules apply from August 2026); (4) minimal-risk (most current AI). An agent used for employment screening or credit decisions is likely high-risk. The Digital Omnibus deferral of high-risk Annex III obligations from August 2026 to 2 December 2027 — provisional as of May 2026 — has since been finalized: the European Parliament endorsed it on 16 June 2026 and the Council gave final approval on 29 June 2026, so it now takes effect on publication in the EU Official Journal rather than as a pending proposal. Annex I products (AI embedded in regulated products) get a separate, later deferral to 2 August 2028.

**CCPA / CPRA (California).** Gives California consumers rights to know, delete, and opt out of sale/sharing of their personal information (oag.ca.gov/privacy/ccpa). Deletion rights extend to training data: the CPPA has signaled that personal data used to train AI models must be deletable on request, which may require model retraining for large foundation models. Automated decision-making that significantly affects consumers (credit, employment, content moderation) requires notification and a meaningful opt-out.

## Tooling

**Microsoft Presidio** (MIT, github.com/microsoft/presidio) — the most widely used open-source PII detection and anonymization library. Two components: Presidio Analyzer (NLP + pattern matching to identify PII spans) and Presidio Anonymizer (applies configurable operators: replace, mask, redact, encrypt). Detects names, emails, phone numbers, credit-card numbers, SSNs, and other entity types. Python; MIT license; latest release 2.2.362 (March 2026). Integrates directly into NeMo Guardrails as a PII-detection backend.

**Google Cloud Sensitive Data Protection** (formerly Cloud DLP) — managed cloud service for discovering, classifying, and de-identifying sensitive data. Includes 200+ built-in infoType detectors, custom infoType support, de-identification transformations (masking, redaction, encryption, tokenization, date shifting), and risk analysis. Can scan BigQuery, Cloud Storage, Datastore, and database content. Docs: cloud.google.com/security/products/sensitive-data-protection

**NVIDIA NeMo Guardrails** (open source, github.com/NVIDIA/NeMo-Guardrails) — a programmable guardrail framework that includes a PII-detection guardrail in its catalog. Supports Presidio-based detection, GLiNER-PII (entity-recognition model), Private AI integration, and GuardrailsAI validators. Applies detection and masking to inputs, LLM outputs, and retrieved content. Cross-link: /resources/agent-guardrails for the broader guardrail pattern.

## Practical checklist

- [ ] Redact PII from all content before it enters the model context window.
- [ ] Redact PII from logs, traces, and observability exports at the exporter layer.
- [ ] Apply data minimization: pass only the fields the current task requires.
- [ ] Use a ZDR / no-training provider tier for any workload processing regulated data.
- [ ] Execute a signed DPA with every provider and sub-processor that handles personal data.
- [ ] Encrypt personal data at rest and in transit for every store in the agent pipeline.
- [ ] Enforce data residency constraints (EU data in EU regions) where required.
- [ ] Set explicit retention limits and deletion schedules for memory stores and logs.
- [ ] Document your lawful basis and purpose limitation for each processing activity.
- [ ] Maintain a deletion-request procedure, even for data in vector indices or model fine-tunes.
- [ ] Route high-risk processing (employment, credit, health) through a human review step.

## Verified sources

- Microsoft Presidio (MIT, GitHub): https://github.com/microsoft/presidio
- Google Cloud Sensitive Data Protection: https://cloud.google.com/security/products/sensitive-data-protection
- NVIDIA NeMo Guardrails — PII detection overview: https://docs.nvidia.com/nemo/guardrails/latest/about/overview.html
- OpenAI enterprise privacy and ZDR: https://openai.com/enterprise-privacy/
- Anthropic Privacy Center: https://privacy.anthropic.com/en/
- GDPR full text (EUR-Lex): https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
- EU AI Act full text (EUR-Lex, Regulation (EU) 2024/1689): https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
- CCPA official page (CA Attorney General): https://oag.ca.gov/privacy/ccpa
- EDPB coordinated enforcement on right to erasure (2025): https://www.edpb.europa.eu/our-work-tools/our-documents/other/coordinated-enforcement-action-implementation-right-erasure_en
- Council final green light, 29 June 2026 (Consilium press release — independently re-verified this cycle; corrects an earlier draft citation that pointed at the 7 May 2026 provisional-agreement press release instead): https://www.consilium.europa.eu/en/press/press-releases/2026/06/29/artificial-intelligence-council-gives-final-green-light-to-simplify-and-streamline-rules/
- Gibson Dunn — EU AI Act Omnibus Agreement, postponed high-risk deadlines: https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines-and-other-key-changes/


Prompt Caching for AI Agents

Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.

Prompt caching lets the provider reuse the KV-cache from a prior request when the leading prefix of a new request matches one it has already processed. Repeated tokens are read at a fraction of the standard input price instead of being recomputed. For agents that send the same system prompt, tool definitions, or reference documents on every loop iteration, caching is the single highest-ROI cost lever available. See /resources/agent-cost-latency-optimization for the broader optimization stack.

## When does caching pay off?

**Caching pays off when:**

- A large, stable block (system prompt, tool schemas, RAG chunks, few-shot examples) repeats across many requests within the TTL window.
- The stable prefix exceeds the provider minimum (roughly 1,024–2,048 tokens depending on provider and model).
- Requests arrive frequently enough that write cost is amortized over multiple cache reads within the TTL.
- Agent loop step count is high — savings compound linearly with the number of calls.

**Caching does not pay off when:**

- Every request has a different prefix (no stable leading block).
- Request volume is very low — a single cache read may not recoup the write premium.
- The TTL expires between requests (e.g., batch jobs spaced hours apart).
- Output tokens dominate cost — caching only discounts input tokens.

## How to trigger caching (by provider)

**Anthropic (Claude)** — manual activation. Place a `cache_control` field on a content block to mark the cache breakpoint (up to 4 breakpoints per request):

```json
{ "type": "text", "text": "...long stable context...",
  "cache_control": { "type": "ephemeral" } }
```

The default TTL is 5 minutes; a 1-hour TTL is available via `"ttl": "1h"`. Confirm a cache hit by reading `cache_creation_input_tokens` and `cache_read_input_tokens` in the response `usage` object. Cache reads are billed at a large discount versus base input; cache writes carry a modest premium (higher for the 1-hour TTL). Verify the current multipliers and per-model minimum cacheable tokens in the pricing docs.

**OpenAI (GPT-4o and later)** — fully automatic, no code change. Caching activates when the prompt is 1,024 tokens or longer; the API caches the longest matching prefix in 128-token increments. Cached counts appear in `usage.prompt_tokens_details.cached_tokens`.

**Google Gemini** — two modes. *Implicit caching* (Gemini 2.5+) is automatic with no code, with a minimum of roughly 2,048 tokens. *Explicit context caching* creates a named cache object with a configurable TTL (default ~60 minutes) referenced by ID; it adds a storage cost per cached-token-hour. Use explicit caching when you need guaranteed reuse.

**DeepSeek** — automatic prefix caching applied when requests share a common prefix from token 0 (no partial mid-input matches). Cache-hit input is priced far below cache-miss; exact thresholds and rates are not consistently documented — verify in your billing dashboard.

## Cross-provider comparison

| Provider | Activation | Min tokens | TTL | Read discount | Notes |
|---|---|---|---|---|---|
| **Anthropic Claude** | Manual (`cache_control` breakpoints) | ~512–4,096 (model-dependent) | 5 min default, 1 h opt-in | Large (cache read ≈ 0.1× base input) | Write carries a premium (higher for 1 h); up to 4 breakpoints |
| **OpenAI GPT-4o+** | Automatic (prefix-based) | 1,024 | ~5–10 min idle (longer for some models) | Significant on cached prefix (verify per model) | Caches in 128-token increments; no explicit controls |
| **Google Gemini 2.5+** | Implicit (auto) or explicit (named cache) | ~2,048 | Implicit: opportunistic; Explicit: ~60 min, configurable | Large on cached read | Explicit caching adds a per-cached-token-hour storage cost |
| **DeepSeek** | Automatic (exact prefix from token 0) | Not officially documented | Not officially documented | Large cache-hit discount | Exact-prefix match only; verify current pricing |

Exact discount percentages and minimum-token thresholds change per model and over time — treat this table as directional and confirm against each provider's current pricing docs (linked below) before relying on a figure.

## Structuring prompts to maximize cache hits

1. **Stable content at the top.** Put the system prompt, tool schemas, and reference documents before any dynamic content — cache matching is prefix-based.
2. **Dynamic content at the tail.** User messages, current task context, and retrieved chunks go after the cache breakpoint.
3. **Keep stable blocks stable.** Any edit to a cached block invalidates the cache; treat cached content as immutable across a session.
4. **Monitor hit rates.** All major providers expose cache-read token counts in the usage response. Track cache_read_tokens / total_input_tokens per endpoint; a low ratio for an agent with a stable system prompt signals missed opportunities.

For prompt-structure principles that interact with caching, see /resources/prompt-context-engineering.

## Stacking with batch discounts

Prompt caching and a batch API (50% off all tokens — see /resources/handling-rate-limits-and-retries) stack independently: a batched request that hits cache pays the batch rate on uncached tokens and the cache-read rate on cached tokens — the cheapest combination for non-latency-sensitive work.

## Verified sources

- Anthropic prompt caching docs: https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- Anthropic pricing (cache write/read multipliers): https://platform.claude.com/docs/en/about-claude/pricing
- OpenAI prompt caching announcement: https://openai.com/index/api-prompt-caching/
- Google Gemini implicit caching announcement: https://developers.googleblog.com/en/gemini-2-5-models-now-support-implicit-caching/
- Google Gemini context caching docs: https://ai.google.dev/gemini-api/docs/caching
- DeepSeek context caching guide: https://api-docs.deepseek.com/guides/kv_cache


Handling LLM Rate Limits (HTTP 429) and Retries for Agents

A practical reference for agent builders: what a 429 means, how to read provider rate-limit headers, exponential backoff with jitter, client-side throttling, and when to use a batch API.

When an LLM provider returns HTTP 429 (Too Many Requests), the calling agent has exceeded one or more rate limits for the current window. **The first thing to do: read the `Retry-After` header (or its provider-specific equivalent) and wait that many seconds before retrying. Do not retry immediately.** Most providers also return per-dimension headers on every successful response, letting you throttle proactively rather than reactively.

## Key facts

- On a 429, the immediate move is to honor the `Retry-After` (or equivalent) header instead of retrying right away, and providers also expose per-dimension headroom on every success so agents can throttle before ever hitting a limit.
- Limits actually stack across at least three windows — raw request-per-minute counts, combined or separately tracked input/output token-per-minute budgets, and a daily ceiling that overrides any remaining per-minute headroom.
- Both providers loosen these ceilings as usage and spend grow, through numbered tiers that advance on account age/cumulative spend or purchased credit — but exact thresholds shift over time, so the practical move is to check the dashboard and request headroom ahead of need rather than react to production errors.
- Randomizing retry delay across the full range up to an exponentially growing cap — while never going below whatever wait time the provider mandated — keeps a fleet of retrying clients from all hammering the API at once.
- Before ever reaching a 429, agents can self-throttle by tracking a local budget from the remaining-headroom headers, capping concurrent in-flight calls, or pre-estimating token cost to hold back requests that would blow the budget.
- Because it draws from a separate quota pool and runs at half the per-token price with same-day turnaround, the batch API suits offline, non-interactive workloads like evals or bulk summarization rather than latency-sensitive agent calls.

## What the rate-limit headers tell you

| Header | OpenAI | Anthropic |
|---|---|---|
| Requests remaining | `x-ratelimit-remaining-requests` | `anthropic-ratelimit-requests-remaining` |
| Tokens remaining | `x-ratelimit-remaining-tokens` | `anthropic-ratelimit-tokens-remaining` |
| Input/output tokens remaining | — | `anthropic-ratelimit-input-tokens-remaining` / `...-output-tokens-remaining` |
| Limit (requests / tokens) | `x-ratelimit-limit-requests` / `...-tokens` | `anthropic-ratelimit-requests-limit` / `...-tokens-limit` |
| Reset time | `x-ratelimit-reset-requests` / `...-tokens` | `anthropic-ratelimit-requests-reset` / `...-tokens-reset` (RFC 3339) |
| Wait before retry | `Retry-After` (on 429) | `retry-after` (on 429) |

Google Gemini exposes `x-ratelimit-remaining` and `retry-after` on the same pattern. Anthropic tracks input and output tokens in separate ITPM and OTPM limits; cached input tokens (cache reads) generally do not count toward ITPM, so prompt caching raises effective throughput without a tier change (see /resources/prompt-caching-for-agents).

## RPM, TPM, and TPD — what each limit covers

**RPM (requests per minute)** — total HTTP calls regardless of payload size; high-frequency, low-token bursts hit this first. The window is typically a rolling 60 seconds.

**TPM (tokens per minute)** — input + output tokens per rolling minute. Long contexts or large system prompts burn TPM quickly even at low RPM. Anthropic enforces separate ITPM and OTPM; OpenAI combines them.

**TPD / RPD (per day)** — a daily ceiling across all per-minute windows; hitting it means waiting for the reset regardless of per-minute headroom. Often the binding constraint for lower tiers running overnight workloads.

## Usage tiers and how limits advance

Both major providers raise your limits as you spend more. OpenAI starts every new project on a Free usage tier (low limits, some models unsupported) and uses numbered usage tiers (Tier 1–5 above it) that advance automatically as cumulative spend and account age cross thresholds; Anthropic uses numbered tiers gated by cumulative credit purchase plus a monthly-invoicing tier with negotiated limits. The exact spend thresholds and per-model RPM/TPM/ITPM numbers change over time and per model — read them from your provider dashboard or the official rate-limit docs rather than hard-coding them. The practical rule at every tier: design for the limit you have today, and request an increase before you need it, not after 429s appear in production.

## Exponential backoff with full jitter

Full jitter — randomizing the delay uniformly between zero and the exponential ceiling — spreads retrying agents across time and prevents synchronized retry storms. Always enforce the provider's `Retry-After` value as a hard floor.

```
MAX_RETRIES = 7
BASE_DELAY  = 1      # seconds
CAP_DELAY   = 60     # seconds

def call_with_backoff(request):
    for attempt in range(MAX_RETRIES):
        response = send(request)
        if response.status != 429:
            return response
        floor = int(response.headers.get("Retry-After", 0))   # provider-mandated floor
        ceiling = min(CAP_DELAY, BASE_DELAY * 2 ** attempt)
        wait = max(floor, random_uniform(0, ceiling))         # full jitter
        sleep(wait)
    raise RateLimitExceeded("max retries reached")
```

The ceiling grows exponentially (1s, 2s, 4s, 8s…) but is capped; the `Retry-After` floor ensures you never retry before the provider permits. Source: aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/.

## Throttling client-side before you hit 429

- **Token bucket** — maintain a local token count from the `*-remaining` headers; subtract estimated cost before each request and sleep until the reset timestamp if it would go negative.
- **Concurrency cap** — bound in-flight requests to a fraction of RPM with a semaphore or async queue.
- **Estimated token pre-check** — tokenize the prompt before sending and queue requests that would exceed remaining headroom.

## When to use a batch API instead

For non-urgent, non-interactive work, both major providers offer a batch API that runs on a separate rate-limit pool (so it does not consume your standard RPM/TPM) and costs **50% less per token**, with results delivered within 24 hours:

- **OpenAI Batch API** — submit a JSONL file of many requests; 50% discount; separate pool. Source: platform.openai.com/docs/guides/batch.
- **Anthropic Message Batches API** — submit up to 100,000 requests per batch; 50% off input and output. Source: platform.claude.com/docs/en/api/rate-limits.

Use a batch API for evals, embedding large corpora, offline summarization, and synthetic-data generation — not for interactive agents that need a result in seconds.

## Provider fallback via a gateway

When a provider is at capacity, an LLM gateway can route to a secondary provider instead of retrying the same endpoint — useful for daily-limit exhaustion or regional outages. See /resources/ai-gateways-llm-routing. For workflow-level retries and idempotency, see /resources/durable-execution-for-agents; for malformed-output retries, see /resources/reliable-tool-calling.

## Verified sources

- OpenAI rate limits guide: https://platform.openai.com/docs/guides/rate-limits
- OpenAI Batch API guide: https://platform.openai.com/docs/guides/batch
- Anthropic rate limits reference: https://platform.claude.com/docs/en/api/rate-limits
- AWS: Exponential Backoff and Jitter: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/