Deploying and Serving LLMs for Agents

Serving-stack reference for teams self-hosting open-weight models for agents: production inference servers, local/dev runtimes, managed GPU endpoints, and key serving concepts — with decision guidance by load profile and verified sources.

Agents make many sequential and parallel model calls. Serving-stack choices determine per-call latency, throughput, cost, and how easily existing OpenAI-compatible agent code can route to self-hosted models. The landscape splits into three tiers: production inference servers you self-host on GPUs, local/dev runtimes for development and edge, and managed/serverless GPU endpoints where someone else runs the hardware.

## Why the serving choice matters for agents

An agent making 20 sequential tool calls at 2 s per call waits 40 s end-to-end. Serving decisions affect all three cost axes: latency per call (time-to-first-token, token throughput), cost (per-token fees vs amortized GPU cost), and integration effort (OpenAI-compatible endpoints drop in; non-compatible ones require adapter code).

Key distinction: agents need **continuous/in-flight batching** — the ability to start processing a new request before the previous one finishes generating. Without it, concurrent agent calls queue behind one another and throughput collapses.

## Key facts

- Choosing where to run an LLM breaks down into three tiers: production servers you run yourself, lighter runtimes meant for local development, and hosted GPU capacity someone else operates — each trading off latency, cost, and integration effort differently.
- Agents making many sequential tool calls are especially latency-sensitive: a chain of 20 calls at 2 seconds each already totals 40 seconds end-to-end, so per-call serving latency compounds quickly.
- Continuous/in-flight batching, which lets a new request join a batch before the prior one finishes, is treated as essential for agents — without it, concurrent calls queue up and throughput collapses.
- vLLM and SGLang lead self-hosted serving, built around paged KV-cache management and prefix/KV-cache reuse respectively, while Hugging Face TGI has moved into maintenance mode with Hugging Face itself pointing new deployments to vLLM or SGLang.
- Nearly every option in the stack, from the big self-hosted servers down to Ollama and llama.cpp, ships an OpenAI-compatible endpoint, which is what lets existing agent code point at self-hosted models with little or no adapter work.
- Reusing cached KV-cache entries for a shared prompt prefix pays off most when an agent's calls keep sending the same boilerplate instructions or retrieved passages over and over.
- Managed/serverless GPU providers split into per-token API aggregators, code-defined serverless GPU platforms, and hyperscaler-managed endpoints.

## Production inference servers (self-hosted)

**vLLM** (Apache 2.0, github.com/vllm-project/vllm) — the dominant open-source LLM serving framework. Core innovations: **PagedAttention** (paged KV-cache management borrowed from OS virtual memory — eliminates memory fragmentation and allows larger batches) and **continuous batching** (new requests join the batch mid-flight without waiting for a full batch to complete). Ships an **OpenAI-compatible HTTP server** out of the box. Supports tensor parallelism across multiple GPUs, speculative decoding, and structured output via XGrammar. 85k+ GitHub stars as of July 2026 (v0.24.0, June 2026 release). Apache 2.0.

**SGLang** (Apache 2.0, github.com/sgl-project/sglang) — high-performance serving framework from UC Berkeley / LMSYS. Core innovation: **RadixAttention** — a radix-tree data structure for automatic, fine-grained **prefix/KV-cache reuse** across requests that share common prefixes (system prompts, few-shot examples, RAG context). Delivers up to 6x higher throughput than alternatives on workloads with shared prefixes. OpenAI-compatible endpoint. v0.5.14 (June 2026); powers 400k+ GPUs in production at xAI, NVIDIA, AMD, LinkedIn.

**Hugging Face TGI** (Apache 2.0, github.com/huggingface/text-generation-inference) — Rust + Python + gRPC inference server used by Hugging Face in production for the Inference API and Hugging Chat. Features continuous batching, tensor parallelism, flash attention, and quantization. OpenAI-compatible Messages API (`/v1/chat/completions`). Note: as of March 2026 TGI is in maintenance mode — Hugging Face recommends vLLM or SGLang for new production deployments.

**NVIDIA TensorRT-LLM + Triton** (Apache 2.0, github.com/NVIDIA/TensorRT-LLM) — NVIDIA's Python API for compiling LLMs into optimized TensorRT engines for NVIDIA GPUs, paired with NVIDIA Triton Inference Server (now part of the NVIDIA Dynamo platform as of March 2025) for serving. Key optimizations: kernel fusion, FP8/INT4 quantization, in-flight batching, and paged KV-caching. Highest throughput on NVIDIA hardware; highest ops complexity. Used by Baseten in production.

**LMDeploy** (Apache 2.0, github.com/InternLM/lmdeploy) — toolkit from the InternLM team for compressing, deploying, and serving LLMs. Two engines: **TurboMind** (C++/CUDA, maximum performance) and **PyTorch** (pure Python, easier to extend). OpenAI-compatible API server via `api_server`. Strong performance on vision-language models. v0.14.0 (June 2026).

## Local / dev runtimes

**Ollama** (MIT, ollama.com) — the simplest way to run open-weight models locally. CLI and REST API (OpenAI-compatible at `http://localhost:11434/v1`). One command to pull and run a model; handles quantization, GPU detection, and memory management automatically. macOS, Linux, Windows. v0.31.2 (July 2026). On Apple Silicon, Ollama runs on an MLX-based inference backend by default as of v0.30.0 (May 2026) — promoted from the March 2026 preview; falls back to the llama.cpp Metal backend automatically below 32GB unified memory.

**llama.cpp** (MIT, github.com/ggml-org/llama.cpp) — the foundational C/C++ LLM inference library. Introduced the **GGUF** model format (all weights + metadata in one portable file). Runs on CPU (with SIMD optimization), NVIDIA GPUs, AMD GPUs, Apple Silicon Metal, and edge hardware. Supports 1.5-bit through 8-bit quantization. Grammar-based constrained generation (GBNF) for structured outputs (see /resources/reliable-tool-calling). Ships an OpenAI-compatible HTTP server. The engine inside LM Studio and the predecessor to many production stacks.

**LM Studio** (lmstudio.ai) — cross-platform GUI application (macOS, Windows, Linux) that wraps llama.cpp and MLX backends behind a model browser, chat interface, and OpenAI-compatible local server. Supports running GGUF and MLX models simultaneously. v0.4.0 (January 2026) added parallel requests with continuous batching and a headless server mode; v0.4.15 (June 2026) added full multi-GPU tensor parallelism for CUDA. Free for personal use. Best fit: dev and prototyping; not designed for multi-tenant production.

**MLX** (MIT, github.com/ml-explore/mlx) — Apple's array framework for Apple Silicon, built around the unified memory architecture (CPU and GPU share the same DRAM). **MLX LM** (the companion package) enables LLM text generation and fine-tuning on-device. MLX leads llama.cpp by 20–87% on models under 14B on Apple Silicon where inference is compute-bound. Apple established MLX as the preferred Apple Silicon inference framework at WWDC 2025.

## Managed / serverless GPU and model endpoints

### Per-token APIs (serverless, shared infrastructure)

- **Together AI** (together.ai) — 200+ open-weight models via a unified serverless API. OpenAI-compatible endpoint. Also offers dedicated GPU endpoints, fine-tuning, and batch inference. Best for variable or bursty loads on open-source models.
- **Fireworks AI** (fireworks.ai) — serverless inference for 400+ models with strong latency optimization (P50 TTFT ~150 ms on Llama 3.3 70B). Per-token pricing; on-demand dedicated GPU endpoints on A100/H100/H200/B200 with per-second billing. Fine-tuning to serverless endpoint.
- **Replicate** (replicate.com) — model marketplace + per-second GPU billing; 50,000+ community models plus curated official models. Convenience-first; acquired by Cloudflare (November 2025), continues as an independent brand with planned Workers AI integration.

### Serverless GPU + code-defined infrastructure

- **Modal** (modal.com) — Python-native serverless GPU cloud. One decorator turns a Python function into a serverless GPU endpoint. GPU memory snapshots (alpha) enable fast model cold starts. gVisor sandbox isolation. $87M Series B (October 2025). Used by OpenAI Agents SDK as an official sandbox execution environment.
- **RunPod** (runpod.io) — GPU cloud with serverless endpoints (pay-per-second) and persistent pods. Competitive per-second pricing; FlashBoot for reduced cold starts. Good for bursty inference workloads and model experimentation.
- **Baseten** (baseten.co) — production-grade model serving platform. TensorRT-LLM Engine Builder for automatic model compilation; Triton-based serving; supports LoRA multi-adapter serving and speculative decoding. Targets teams that need maximum NVIDIA GPU utilization.
- **Anyscale** (anyscale.com) — LLM inference on Ray Serve + vLLM. Serverless and dedicated endpoints for popular open-source models. Strong for teams already on the Ray ecosystem.

### Hyperscaler managed endpoints

- **Amazon Bedrock** (aws.amazon.com/bedrock) — managed foundation model layer on AWS. Per-token API access to open-weight models (Llama 4, Mistral, Titan, and others) plus proprietary models via AWS IAM and CloudTrail governance.
- **Azure AI Foundry** (azure.microsoft.com) — Microsoft's managed model platform (formerly Azure AI Studio). Serverless per-token APIs and dedicated managed compute for open-weight models (Llama, Mistral, Phi) plus OpenAI and partner models. Azure-native identity and governance.
- **Google Vertex AI Model Garden** (cloud.google.com/vertex-ai) — Google Cloud managed model endpoints. Per-token APIs for Gemini, open-weight models (Llama, Mistral, Gemma), and partner models. GCP IAM governance.

## Key serving concepts

- **Continuous / in-flight batching** — new requests join the active generation batch immediately, without waiting for a full batch to complete. Essential for agent workloads with concurrent calls. Implemented in vLLM, SGLang, TGI, TensorRT-LLM, and LMDeploy.
- **Paged attention / KV-cache management** — allocates KV-cache in fixed-size pages to eliminate fragmentation, enabling larger effective batch sizes. vLLM's PagedAttention is the canonical implementation; SGLang's RadixAttention extends this with prefix sharing.
- **Prefix / prompt caching** — reuses KV-cache entries for shared prompt prefixes across requests. High-value for agents with identical system prompts or RAG context blocks repeated across calls. See /resources/agent-cost-latency-optimization for provider-level caching.
- **Quantization** — reduces model weight precision (FP16 → INT8 → INT4 → FP8, etc.) to cut VRAM and increase throughput at a small quality cost. GGUF quantization in llama.cpp; AWQ/GPTQ/FP8 in vLLM and TGI. See /resources/open-weight-models-for-agents.
- **Tensor / pipeline parallelism** — splits a model across multiple GPUs. Tensor parallelism splits weight matrices; pipeline parallelism splits layers. Required for models that exceed single-GPU VRAM. All production servers support tensor parallelism.
- **Structured output / grammar support** — constrains token generation to a JSON Schema or BNF grammar at the serving layer. vLLM and SGLang use XGrammar; llama.cpp uses GBNF. See /resources/reliable-tool-calling.

## Decision guidance

| Scenario | Recommended tier |
|---|---|
| Variable / bursty agent load; no GPU ops | Per-token API (Together AI, Fireworks, Bedrock) |
| High steady-volume or data-residency requirement | Self-hosted vLLM or SGLang on dedicated GPU |
| NVIDIA-maximum performance (H100/H200/B200) | TensorRT-LLM + Triton (via Baseten or self-hosted) |
| Dev / local testing | Ollama or LM Studio |
| Apple Silicon on-device / edge | MLX LM or Ollama (MLX backend preview) |
| Low-resource / CPU-only / air-gapped | llama.cpp |

Cross-links: for gateway/routing across providers see /resources/ai-gateways-llm-routing; for cost and latency optimization see /resources/agent-cost-latency-optimization; for open-weight model selection see /resources/open-weight-models-for-agents.

## Verified sources

- vLLM GitHub (Apache 2.0): https://github.com/vllm-project/vllm
- vLLM docs: https://docs.vllm.ai/en/latest/
- vLLM releases (v0.24.0, June 2026): https://github.com/vllm-project/vllm/releases
- SGLang GitHub (Apache 2.0): https://github.com/sgl-project/sglang
- SGLang docs: https://docs.sglang.io/
- SGLang v0.5.14 release notes: https://github.com/sgl-project/sglang/releases/tag/v0.5.14
- Hugging Face TGI GitHub (Apache 2.0, maintenance mode): https://github.com/huggingface/text-generation-inference
- TGI Messages API (OpenAI-compatible): https://huggingface.co/docs/text-generation-inference/en/messages_api
- NVIDIA TensorRT-LLM GitHub (Apache 2.0): https://github.com/NVIDIA/TensorRT-LLM
- TensorRT-LLM Triton backend: https://github.com/triton-inference-server/tensorrtllm_backend
- LMDeploy GitHub (Apache 2.0): https://github.com/InternLM/lmdeploy
- LMDeploy releases (v0.14.0, June 2026): https://github.com/InternLM/lmdeploy/releases
- LMDeploy OpenAI-compatible server: https://lmdeploy.readthedocs.io/en/latest/llm/api_server.html
- Ollama blog: https://ollama.com/blog
- Ollama MLX backend announcement: https://ollama.com/blog/mlx
- Ollama releases (v0.31.2, July 2026): https://github.com/ollama/ollama/releases
- llama.cpp GitHub (MIT): https://github.com/ggml-org/llama.cpp
- LM Studio homepage: https://lmstudio.ai/
- LM Studio 0.4.15 changelog: https://lmstudio.ai/changelog/lmstudio-v0.4.15
- MLX GitHub (MIT, Apple): https://github.com/ml-explore/mlx
- Together AI serverless inference: https://docs.together.ai/docs/serverless/models
- Fireworks AI homepage: https://fireworks.ai/
- Baseten inference stack guide: https://www.baseten.co/resources/guide/the-baseten-inference-stack/
- Modal homepage: https://modal.com/
- RunPod homepage: https://www.runpod.io/
- Replicate homepage: https://replicate.com/
- Anyscale LLM inference: https://www.anyscale.com/use-case/llm-online-inference
- Amazon Bedrock homepage: https://aws.amazon.com/bedrock/
- Azure AI Foundry homepage: https://azure.microsoft.com/en-us/products/ai-foundry/
- Google Vertex AI Model Garden: https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/explore-models


Prompt and Context Engineering for Agents

From crafting a single prompt to managing everything an agent sees across a trajectory: system-prompt design, context-window management, failure modes, and a high-leverage checklist.

## Key facts

- Curating everything visible to a model over a multi-turn trajectory — system prompt, tools, retrieved data, memory, history — is a broader discipline than writing one good instruction, and both Anthropic and Cognition treat it as the higher-leverage skill.
- Agents tend to fail from poor information architecture spread across the whole context window rather than from a single badly worded prompt.
- A well-structured system prompt covers five layers: role and objective, tool-use instructions, constraints/guardrails, output format, and a middle-ground calibration that is precise enough to shape what the agent does without being so rigid it can't handle a case the author didn't enumerate.
- Core prompting techniques include few-shot examples, structured delimiters that separate instruction from data, typed tool/response schemas, explicit chain-of-thought/planning steps, and reaching for a tool instead of encoding complex logic in prose.
- Agent-specific context-window management relies on periodic compaction/summarization, just-in-time retrieval instead of loading everything upfront, separate memory tiers, per-sub-agent context isolation, and prompt caching of a stable prefix.
- Known failure modes include context rot (recall degrading as the window grows), a lost-in-the-middle effect where content sitting between the start and end of a lengthy prompt gets under-weighted, adversarial context poisoning, and clashing instructions from different segments.
- Prompts and context configurations are best treated as versioned artifacts measured with evals, and validated against a second, different model before anyone calls them stable.

## From prompt engineering to context engineering

Prompt engineering is crafting a single instruction. Context engineering is managing the entire token window an agent sees across a multi-step trajectory: system prompt, tool definitions, retrieved data, memory outputs, and accumulated message history. Anthropic defines it as "the set of strategies for curating and maintaining the optimal set of tokens during LLM inference." Cognition (Devin) calls it "the #1 job of engineers building AI agents."

The framing shift matters because agents fail not from a bad one-liner prompt but from poor information architecture across the whole window.

## System-prompt design for agents

A well-structured system prompt typically covers five layers:

1. **Role + objective** — who the agent is and what success looks like.
2. **Tool-use instructions** — when to call which tool, and when not to.
3. **Constraints and guardrails** — what the agent must never do.
4. **Output format** — schema, tone, length, and structure of the response.
5. **"Right altitude" principle** — specific enough to steer behavior; general enough to generalize to cases not enumerated. Avoid exhaustive rule lists; prefer canonical examples.

Anthropic's guidance: if a human engineer cannot definitively say which tool to use in a given situation, an agent cannot either — keep the toolset minimal and curated.

## Core prompt-engineering techniques

- **Clear instructions with few-shot examples** — show desired behavior on representative cases; a small number of diverse canonical examples outperforms a long rule list.
- **Structured delimiters** — XML tags (`<task>`, `<context>`, `<constraints>`) or markdown sections isolate logical regions and reduce ambiguity about what is instruction versus data.
- **Tool and response schemas** — typed JSON schemas for tool inputs and agent outputs eliminate ambiguity. See /resources/reliable-tool-calling.
- **Chain-of-thought / explicit planning** — instruct the agent to reason before acting (`<thinking>` block or a planning step) to reduce shallow-reasoning errors on multi-step tasks.
- **Tools instead of long instructions** — when a behavior requires complex retrieval or calculation, provide a tool rather than trying to encode the logic in prose.

## Context-window management (the agent-specific layer)

For single-turn LLM calls, context is static. For agents the window is dynamic and grows across steps. Key management strategies:

- **Compaction / summarization** — periodically replace accumulated history with a compact summary before the window fills. The Claude API exposes a token-efficient compaction mode.
- **Just-in-time retrieval** — pull only the most relevant context at each step instead of loading everything upfront. See /resources/rag-retrieval-for-agents.
- **Memory tiers** — distinguish in-context working memory from external persistent stores. See /resources/agent-memory-context.
- **Context isolation per sub-agent** — give each sub-agent a scoped window so noisy tool outputs or unrelated history cannot distract it. Per Cognition: share full agent traces, not just individual messages, when coordination is needed.
- **Prompt caching** — a stable, long system-prompt prefix is cached by major providers, making a rich prefix cheap to repeat across turns. See /resources/agent-cost-latency-optimization.

## Failure modes to know

| Failure mode | Description | Mitigation |
|---|---|---|
| Context rot | Recall degrades as window length grows | Compaction; just-in-time retrieval |
| Lost-in-the-middle | Model under-weights information in the middle of a long context | Put high-priority context at the start or end; summarize the middle |
| Context poisoning | Adversarial data injected into retrieved content or tool outputs | Treat all external content as untrusted data, not instructions. See /resources/agentic-security-checklist |
| Context clash / distraction | Conflicting instructions from different context segments | Use structured delimiters; isolate sub-agent windows |

## Evaluation and iteration

Treat prompts and context configurations as versioned artifacts (store in version control, review in PRs). Measure changes with evals before deploying — see /resources/evaluating-ai-agents. Avoid over-fitting to one model's quirks: test on at least two models before calling a prompt stable.

## High-leverage context-engineering checklist

- [ ] System prompt has all five layers (role, tools, constraints, format, examples).
- [ ] Toolset is minimal: only tools the agent needs for this task.
- [ ] At least two diverse few-shot examples per non-trivial behavior.
- [ ] Structured delimiters separate instruction, data, and history regions.
- [ ] Tool schemas and response schemas are typed (JSON Schema or equivalent).
- [ ] History compaction or summarization strategy is defined.
- [ ] Retrieval is just-in-time, not full-corpus-upfront.
- [ ] Context isolation is applied to sub-agents doing independent work.
- [ ] Prompt is stored in version control and covered by evals.
- [ ] Tested on more than one model before treating as stable.

## Verified sources

- Anthropic Engineering — "Effective context engineering for AI agents": https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- Cognition — "Don't Build Multi-Agents" (context engineering principles section): https://cognition.ai/blog/dont-build-multi-agents
- Anthropic Docs — Prompt engineering overview: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview
- OpenAI — Prompt engineering guide: https://platform.openai.com/docs/guides/prompt-engineering/strategy-write-clear-instructions


Agent Reasoning and Design Patterns

The canonical single-agent reasoning and acting loops: ReAct, Chain-of-Thought, Plan-and-Solve, ReWOO, Reflexion, Tree-of-Thoughts, and Self-Consistency — what each is, when to use it, and tradeoffs.

Every agent implementation rests on the same substrate: a loop that perceives input, reasons about what to do, executes an action (usually a tool call), observes the result, and repeats until a termination condition is met. The named patterns below are systematic ways of structuring that loop. Choose the simplest pattern that meets your correctness and cost requirements.

## The core loop

**Perceive → Reason → Act → Observe → repeat.**

Each turn the agent reads its context window (system prompt, history, tool results), produces reasoning or a plan, calls a tool or returns a final answer, and appends the result to context. Two decisions recur in every pattern: (1) *which tool to call* (driven by the reasoning step and the available tool schema); (2) *when to stop* (goal-satisfaction check, max-iteration guard, or explicit DONE signal). Error recovery follows a retry-then-replan ladder: retry the same call on transient failures, replan on semantic failures, escalate to the caller on persistent failures.

## Named patterns

### ReAct — reason + act interleaved

The model emits a *Thought* (free-form reasoning trace), then an *Action* (tool call), then an *Observation* (tool result), cycling until it can emit a final answer. Interleaving reasoning with actions keeps the scratchpad grounded: each thought is informed by the most recent real observation. **When to use:** general-purpose tool-using agents, short-to-medium horizon tasks. **Tradeoff:** reasoning tokens paid on every turn; long loops accumulate context fast.

### Chain-of-Thought (CoT)

The model reasons step-by-step in natural language before producing an answer, without interleaved tool calls. Strong for arithmetic, logic, and multi-step inference when all information is already in context. **Limit:** no mechanism to retrieve missing information or recover from wrong intermediate steps. **Reasoning-native models** (OpenAI's o-series, Claude's extended thinking, Gemini's thinking variants) generate this step-by-step trace internally as part of inference, so explicitly prompting "think step by step" is often redundant and can degrade output on these models — prefer concise, direct prompts and let the model's built-in reasoning run (WebSearch-convergence across multiple vendor prompting guides, as of mid-2026). Explicit CoT prompting still applies to non-reasoning/base chat models.

### Plan-and-Execute / Plan-and-Solve

Phase 1: produce an explicit multi-step plan. Phase 2: execute each step, optionally re-planning when a step fails. Separating planning from execution lets the agent reason globally before committing to actions. **When to use:** long-horizon tasks where the full sequence of steps can be enumerated upfront. **Tradeoff:** plan may go stale if early steps produce unexpected results; requires a replanning trigger.

### ReWOO — reasoning without observation

All tool calls are planned upfront in a single reasoning pass, with placeholders for their outputs. The planner runs once; an executor runs the tool calls in sequence, substituting real results for placeholders; a solver synthesizes the final answer. Because the LLM is not re-invoked between tool calls, token consumption is dramatically lower for deterministic tool sequences. **When to use:** tasks with predictable, independent tool calls. **Tradeoff:** cannot adapt mid-sequence if an early result changes what later calls should be.

### Reflexion — self-reflection and retry

After a failed or low-quality attempt, the agent critiques its own output in natural language ("verbal reinforcement"), stores the critique in an episodic memory buffer, and retries with that critique in context. No weight updates are required — the feedback loop is entirely in-context. **When to use:** tasks with a verifiable success signal (unit tests, factual checks) where correctness matters more than token cost. **Tradeoff:** multiple full attempts multiply cost; critique quality depends on the model's self-awareness.

### Tree-of-Thoughts (ToT)

At each step, the agent generates multiple candidate reasoning branches, scores or votes on them, and searches the tree (BFS or DFS) for the most promising path. Generalizes Chain-of-Thought from a single chain to a search over reasoning. **When to use:** problems requiring deliberate exploration where greedy reasoning routinely fails (planning puzzles, creative tasks with many candidate solutions). **Tradeoff:** compute cost scales with branching factor and depth; impractical for latency-sensitive tasks.

### Self-Consistency

Sample multiple independent reasoning paths (temperature > 0), then select the answer that appears most often across paths (majority vote). No tree search or explicit scoring — diversity is obtained by sampling, consistency is the signal. **When to use:** arithmetic and commonsense reasoning where a correct answer is unique and verifiable. **Tradeoff:** cost scales linearly with the number of samples; diminishing returns beyond ~10–20 paths.

## Tool-use and stopping

- **Tool selection:** driven by the tool schema in the system prompt; structured output (JSON function-call format) reduces mis-calls. See /resources/reliable-tool-calling.
- **Termination:** explicit DONE / final-answer token, goal-satisfaction check in code, or max-iteration guard (always set one to prevent runaway loops).
- **Error recovery ladder:** (1) retry identical call on transient/network errors; (2) replan the current step on semantic errors; (3) escalate to the caller on persistent failures. Log every failure with its tool name and arguments.

## Reliability techniques

- **Explicit planning** reduces mid-sequence confusion on long-horizon tasks.
- **Structured scratchpads** (labeled Thought / Action / Observation blocks) prevent the model from conflating reasoning with output.
- **Verification / self-check steps** after each action ground the agent in what actually happened, not what it expected.
- **Grounding in tool results** means: never reason forward from an assumed tool output; always wait for the real observation.
- More reasoning steps cost more tokens and add latency — see /resources/agent-cost-latency-optimization.
- Pattern choice interacts with how you structure the system prompt and history — see /resources/prompt-context-engineering.

## Decision guide

1. **Start with ReAct + well-typed tools.** It covers the majority of tool-using tasks with minimal complexity. Add structured output / function-calling format to reduce mis-calls.
2. **Add explicit planning** (Plan-and-Execute or ReWOO) when the task is long-horizon and the step sequence is predictable.
3. **Add reflection / verification** (Reflexion, self-check steps) when correctness matters more than cost and a success signal exists.
4. **Use self-consistency or ToT** only when single-pass reasoning demonstrably fails on your task and you can afford the compute.
5. **Escalate to multi-agent** only when a single agent genuinely cannot hold the required context, tools, or specialization — see /resources/multi-agent-orchestration-patterns.

Cross-links: /resources/reliable-tool-calling · /resources/evaluating-ai-agents · /resources/agent-cost-latency-optimization · /resources/prompt-context-engineering · /resources/multi-agent-orchestration-patterns

## Verified sources

- ReAct (arXiv:2210.03629) — Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models", 2022: https://arxiv.org/abs/2210.03629
- Reflexion (arXiv:2303.11366) — Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning", NeurIPS 2023: https://arxiv.org/abs/2303.11366
- Tree-of-Thoughts (arXiv:2305.10601) — Yao et al., "Tree of Thoughts: Deliberate Problem Solving with Large Language Models", NeurIPS 2023: https://arxiv.org/abs/2305.10601
- ReWOO (arXiv:2305.18323) — Xu et al., "ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models", 2023: https://arxiv.org/abs/2305.18323
- Self-Consistency (arXiv:2203.11171) — Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models", 2022: https://arxiv.org/abs/2203.11171
- Plan-and-Solve (arXiv:2305.04091) — Wang et al., "Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models", ACL 2023: https://arxiv.org/abs/2305.04091


Durable Execution for Long-Running Agents

Vendor-neutral reference on durable execution: event logs, replay determinism, idempotency, retries, and human-in-the-loop pause/resume — plus a cross-vendor survey and tradeoffs guide for Temporal, Restate, DBOS, Inngest, Step Functions, Azure Durable Functions, Cloudflare Workflows, GCP Workflows, LangGraph, and OpenAI Agents SDK.

A long-running agent that cannot survive a crash, a restart, or a long wait for human approval is a liability in production. Durable execution is the programming model that solves this. This reference explains the core concepts vendor-neutrally, surveys the leading engines, and maps them to a "when to use which" decision guide.

## Key facts

- Durable execution logs every meaningful workflow step to persistent storage first, so that after a crash or restart the engine can rebuild in-process state from those stored records instead of re-running the same real-world actions.
- Because resuming re-runs the orchestration function, that function has to yield the same commands given the same recorded inputs; anything non-deterministic — clock reads, randomness, direct LLM or API calls — has to be pushed into a separately recorded unit rather than live inside the workflow logic itself.
- A crash can cause a step that was mid-flight to be attempted a second time, so side-effecting calls need a stable key built from the run and step identity, not from time or a fresh random value, so a duplicate call lands as a no-op.
- Parking a workflow on an external signal — an approval, a webhook, an event — holds no compute meanwhile and restores full state the instant the signal shows up, unlike an actively polling or sleeping process.
- Current engines fall into four broad shapes: full-replay engines that rebuild state by re-executing the event history (Temporal, Azure Durable Functions), journal-based engines that short-circuit already-finished steps (Restate, DBOS, Inngest), workflow definitions where a cloud provider — not application code — owns and advances the execution state (AWS, GCP), and Durable-Object-hosted code (Cloudflare) — with LangGraph's checkpointer and the Agents SDK's session store offering a lighter, single-graph or single-session alternative rather than a full cross-service orchestrator.
- Picking one mostly comes down to existing platform commitment (native engines on AWS, Azure, GCP, or Cloudflare), a self-host-vs-managed tradeoff among the portable code-first options (Temporal, Restate, DBOS versus Inngest), and whether a declarative DSL's tighter service hooks are worth giving up plain annotated functions.

## Core concepts

### Persisted event log / checkpoint

A durable execution engine records every meaningful step of a workflow — LLM call, tool invocation, timer, signal received — to a persistent log (or checkpoint store) before moving on. If the process crashes or is restarted, the engine replays the log to reconstruct in-memory state exactly where execution left off. No step that was recorded is re-executed against the real world; its stored result is injected instead.

### Determinism / replay constraints

For replay to be safe, *workflow code* (the orchestration logic) must be deterministic: given the same sequence of recorded inputs, it must produce the same sequence of scheduled commands. This means workflow code must not call wall-clock time, random-number generators, external APIs, or LLMs directly. Those calls are non-deterministic and belong in a separate unit — called an *activity*, *step*, or *handler* depending on the engine — that is executed outside the replay path, recorded once, and replayed from the record on retry.

### Idempotency

Because a crashed workflow may retry a step that was already in-flight, side-effecting tool calls (writes to a database, calls to a payment API, sends to a message bus) must be idempotent. The standard pattern is to derive a stable idempotency key from the workflow run ID and step index — not from wall-clock time or a random UUID — and pass it to the downstream service so that a duplicate call is a no-op. See also /resources/reliable-tool-calling for tool-call reliability patterns.

### Retries, timeouts, and backoff

All major engines provide per-step retry policies: maximum attempts, initial interval, backoff multiplier, and a non-retryable error list. The recommended distinction: *transient* failures (network timeout, rate limit, 5xx) are retried with backoff; *terminal* failures (4xx bad-request, business logic error) escalate immediately. Unbounded retries mask bugs; always set a maximum. Check current docs for default retry parameters, which vary by engine and are subject to change.

### Durable human-in-the-loop (pause/resume)

A durable engine can suspend a workflow at any step and wait for an external signal — a human approval, a webhook, an event — for an arbitrarily long duration without holding a live process or consuming compute. When the signal arrives the workflow resumes from exactly where it paused, with full state intact. This is architecturally distinct from a polling loop or a sleeping thread: the workflow is durably parked, not actively running.

## Cross-vendor survey

### Architectural models

Four broad models exist across current engines:

| Model | Description | Examples |
|---|---|---|
| Event-sourcing / full replay | Workflow function is re-executed from start on every resume; each step is compared against the event history and short-circuited | Temporal, Azure Durable Functions |
| Durable-state-store / code-first | Journal records step outcomes; replay injects stored results without full re-execution of orchestration logic | Restate, DBOS (Postgres-backed), Inngest (memoization) |
| Managed state machine | Workflow is a declarative YAML/JSON state machine; engine manages state externally | AWS Step Functions (ASL), GCP Workflows |
| Code-on-Durable-Objects | Worker code runs on a per-workflow Durable Object; steps are persisted as the object's durable state | Cloudflare Workflows |

### Temporal

Event-sourcing model. The Temporal service maintains a durable event history per workflow execution. On resume, the SDK re-executes the workflow function from start, comparing each generated command against the history; recorded activity results are injected without re-running. Non-deterministic operations (LLM calls, tool I/O, timers, randomness) must be wrapped as Activities. Human-in-the-loop is implemented with Signals: a workflow blocks on a signal condition while consuming no compute until the signal arrives. Open-source server; self-hostable or managed via Temporal Cloud.

### Azure Durable Functions (Durable Task Framework)

Event-sourcing model, closely analogous to Temporal. Orchestrator functions checkpoint progress at every `await`/`yield`, saving history to a durable storage backend (Azure Storage by default; MSSQL and other providers also supported). On resume, the framework replays the orchestrator from start and injects previously recorded activity results. Orchestrator code must be deterministic; side effects go in Activity functions. External events (`waitForExternalEvent`) implement human-in-the-loop pause/resume. Available as a first-party Azure Functions extension.

### Restate

Durable-state-store / code-first model. Restate tracks execution in a per-invocation journal on its server. If a handler crashes, Restate replays the journal — skipping completed steps by returning their stored results — and resumes from the failure point. Idempotency is built in via idempotency-key headers; duplicate requests are deduplicated automatically. Human-in-the-loop is supported via durable promises / suspension: a handler suspends and is resumed when an external call provides the result. Open-source; self-hostable or managed via Restate Cloud.

### DBOS

Durable-state-store model backed by Postgres. DBOS Transact is a library — now spanning Python, TypeScript, Go, Java, and Kotlin (WebSearch-corroborated via dbos.dev and the dbos-inc GitHub org; not independently re-fetched this cycle, see Verified sources) — that annotates workflows and steps; execution state is stored in the application's own Postgres database. If a workflow is interrupted, it automatically resumes from the last completed step on restart. Designed to be added to an existing application without a separate orchestration server. Open-source library; also available as a managed cloud service.

### Inngest

Durable-state-store / step-memoization model. Functions are broken into `step.run()` units; each step's result is persisted after it completes. On retry the function re-executes, but completed steps return their memoized results immediately rather than re-running. `step.waitForEvent()` implements human-in-the-loop: the function suspends until a matching external event arrives, consuming no compute while waiting. Historically a managed-only service with an open-source SDK; alongside its 1.0 release the Inngest server itself also shipped an official self-hosting path (single binary or Docker, SQLite by default with optional Postgres/Redis for production) — WebSearch-corroborated, 5 sources agreeing (inngest.com's own docs and blog, the official inngest-helm chart repo, and two independent third-party write-ups); not independently re-fetched this cycle due to a session-wide WebFetch outage, see Verified sources.

### AWS Step Functions

Managed state machine model. Workflows are defined in Amazon States Language (ASL), a JSON/YAML state machine definition. The Step Functions service manages all state externally; application code runs in Lambda or other compute only for task states. Standard Workflows are durable up to one year with an exactly-once execution model per state. No replay of application code: state is always held by the service, not reconstructed by replaying code. Integrates natively with most AWS services via optimized integrations. Human-in-the-loop via callback patterns with task tokens (`.waitForTaskToken`).

### GCP Workflows

Managed state machine model. Workflows are defined in YAML or JSON using Google's Workflows syntax. The service manages execution state; a workflow can hold state, retry, poll, or wait for up to one year as documented. Human-in-the-loop is supported via callback endpoints: the workflow pauses and waits for an external HTTP callback to resume it. Serverless; no charges while idle.

### Cloudflare Workflows

Code-on-Durable-Objects model. Each workflow instance runs on a dedicated Cloudflare Durable Object; step state is persisted as the object's durable storage. Step primitives: `step.do()` (execute with automatic retry), `step.sleep()` / `step.sleepUntil()` (hibernates the object — no compute consumed during sleep), and `step.waitForEvent()` (suspends until an external event arrives). Tightly integrated with the Cloudflare Workers ecosystem. Reached general availability.

### LangGraph checkpointers

Framework-level checkpoint mechanism within LangGraph (part of LangChain). A checkpointer persists graph state after every node execution to a configurable backend (in-memory, SQLite, Postgres, and others as documented). Execution is tracked by `thread_id`; resuming with the same `thread_id` reloads the last checkpoint. Human-in-the-loop is implemented via `interrupt()`: calling interrupt inside a node raises a `GraphInterrupt`, saves state, and surfaces the interrupt value to the caller; the graph resumes when the caller re-invokes with a `Command` containing the human's response. This is a lighter-weight mechanism than a full durable execution engine — it provides fault tolerance and HITL within a single agent graph, not cross-process workflow orchestration.

### OpenAI Agents SDK sessions

Session-persistence layer within the Agents SDK. A Session stores conversation history across agent runs to a configurable backend (SQLite, Redis, MongoDB, SQLAlchemy-compatible stores, OpenAI Conversations API, and others as documented). Before each run, the runner prepends session history to the input; after each run, new items are persisted. This is *memory continuity*, not durable execution: sessions do not guarantee transactional recovery from infrastructure failures mid-run. Suitable when the SDK's built-in session management is sufficient and full crash-resume semantics are not required.

## When to use which

| Situation | Recommended approach |
|---|---|
| Already on AWS; need durable multi-step agent workflows | AWS Step Functions (Standard Workflows) — native integration, zero extra infra |
| Already on Azure | Azure Durable Functions — first-party, event-sourcing model, supports long-running orchestrations |
| Already on GCP | GCP Workflows — managed state machine, callback-based HITL |
| Already on Cloudflare Workers | Cloudflare Workflows — co-located with edge compute, Durable Objects-backed |
| Need portable, code-first durability; want to self-host | Temporal (mature, large ecosystem), Restate (lighter footprint, suspension-native), or DBOS (Postgres-only dependency) |
| Managed serverless, code-first, TypeScript/JavaScript-first | Inngest — step memoization, managed infra, `waitForEvent` HITL (an official, newer self-hosting path now also exists) |
| Already using LangGraph; need HITL and checkpoint-based fault tolerance within a graph | LangGraph checkpointers + `interrupt()` — no extra service needed |
| Using OpenAI Agents SDK; need memory continuity across sessions but not crash-resume semantics | Agents SDK Sessions — simplest path; add a dedicated engine if mid-run durability is required |

Key tradeoffs to weigh:

- **Replay vs. state-store:** event-sourcing/replay engines (Temporal, Azure DF) require strict workflow determinism; state-store engines (Restate, DBOS, Inngest) are often more forgiving but still disallow non-deterministic branching on replay.
- **Self-host vs. managed:** Temporal, Restate, and DBOS are self-hostable but add operational burden; Inngest added an official self-hosting path alongside its 1.0 release but is newer and less proven in that mode; cloud-native engines offload ops at the cost of platform lock-in.
- **Granularity:** declarative state machines (Step Functions, GCP) offer fine-grained service integration but require workflow logic to fit a declarative model; code-first engines let you write ordinary functions with SDK annotations.
- **Ecosystem fit:** LangGraph checkpointers and Agents SDK sessions add no external service dependency, making them the lowest friction entry point when you are already in those frameworks — but they are not substitutes for a full orchestration engine when cross-service durability or complex retry/compensation logic is needed.

For reliability patterns at the tool-call level, see /resources/reliable-tool-calling. For multi-agent orchestration patterns that interact with durable workflows, see /resources/multi-agent-orchestration-patterns. For observability inside long-running agent workflows, see /resources/agent-observability. For comparing the broader framework landscape, see /resources/agent-frameworks-compared. For cost and latency considerations in long agent loops, see /resources/agent-cost-latency-optimization. For guardrails on autonomous actions that durable workflows may take, see /resources/agent-guardrails. ChangeGamer applies the same resume-from-persisted-state idea at a coarser, cycle-level grain (a version-controlled BACKLOG and JOURNAL, not an in-process checkpoint log) — see the premium resource [How ChangeGamer Runs Itself](/resources/autonomous-operations).

## Verified sources

- Temporal — Workflow Execution overview: https://docs.temporal.io/workflow-execution
- Temporal — Event History: https://docs.temporal.io/encyclopedia/event-history
- Temporal — Workflow message passing (Signals): https://docs.temporal.io/encyclopedia/workflow-message-passing
- Temporal — Workflow Definition (determinism): https://docs.temporal.io/workflow-definition
- Temporal — Human-in-the-Loop tutorial: https://learn.temporal.io/tutorials/ai/building-durable-ai-applications/human-in-the-loop/
- Restate — Durable Execution concepts: https://docs.restate.dev/concepts/durable_execution/
- DBOS — DBOS Transact open-source library: https://www.dbos.dev/dbos-transact
- DBOS — Go/Java SDK repos (WebSearch-corroborated, not independently re-fetched this cycle): https://github.com/dbos-inc/dbos-transact-golang, https://github.com/dbos-inc/dbos-transact-java
- Inngest — How functions are executed (step memoization): https://www.inngest.com/docs/learn/how-functions-are-executed
- Inngest — step.waitForEvent reference: https://www.inngest.com/docs/reference/typescript/functions/step-wait-for-event
- Inngest — Self-hosting docs and 1.0 announcement (WebSearch-corroborated, not independently re-fetched this cycle): https://www.inngest.com/docs/self-hosting, https://www.inngest.com/blog/inngest-1-0-announcing-self-hosting-support
- AWS Step Functions — What is Step Functions: https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html
- AWS Step Functions — State machines: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-statemachines.html
- Azure — Durable Orchestrations overview: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestrations
- Cloudflare — Workflows durable execution: https://developers.cloudflare.com/agents/api-reference/durable-execution/
- Cloudflare — Workflows GA announcement: https://blog.cloudflare.com/workflows-ga-production-ready-durable-execution/
- GCP Workflows — Overview: https://docs.cloud.google.com/workflows/docs/overview
- LangGraph — Persistence and checkpointers: https://docs.langchain.com/oss/python/langgraph/persistence
- LangGraph — Interrupts (human-in-the-loop): https://docs.langchain.com/oss/python/langgraph/interrupts
- LangGraph — Durable execution: https://docs.langchain.com/oss/python/langgraph/durable-execution
- OpenAI Agents SDK — Sessions: https://openai.github.io/openai-agents-python/sessions/


MCP Primitives: Resources, Prompts, Sampling, and Elicitation

Deep reference on the six MCP capability primitives beyond tools — who controls each, the exact JSON-RPC method names, and when to use Resources vs Tools — verified against the 2025-06-18 and 2025-11-25 spec revisions.

MCP defines six capability primitives in two groups: server-side (Tools, Resources, Prompts) and client-side (Sampling, Roots, Elicitation). Each primitive has a designated controlling party — model, application, or user — which determines how it is invoked and why the separation matters for safety. This page goes deep on the five primitives beyond Tools, which are covered separately in /resources/mcp-vs-a2a and /resources/mcp-server-discovery.

## Control model overview

| Primitive | Side | Controlled by | Who decides to invoke it |
|---|---|---|---|
| Tools | Server | Model | The LLM autonomously decides to call a tool |
| Resources | Server | Application | The host app decides what context to expose and inject |
| Prompts | Server | User | The user explicitly selects a prompt template |
| Sampling | Client | Server (asks client) | The server requests an LLM call from the host |
| Roots | Client | Application | The client app declares filesystem/URI boundaries |
| Elicitation | Client | Server (asks user via client) | The server requests structured user input mid-session |

The three-way split (model / app / user) is not cosmetic. It maps directly to who bears responsibility for an action and what approval is required before it executes. See /resources/agentic-security-checklist and /resources/mcp-server-authentication for the security implications.

## Server-side primitives

### Resources (application-controlled)

Resources are read-only data that the host application exposes to the model as context. Unlike tools, the model never decides to fetch a resource on its own — the application controls what is offered and when it is injected into context.

A resource has a URI (e.g. `file:///workspace/README.md`, `db://customers/42`) and returns either text or binary blob content.

**JSON-RPC methods (verified against 2025-11-25 schema):**

- `resources/list` — enumerate all available resources
- `resources/read` — fetch the content of a specific resource by URI
- `resources/templates/list` — list URI templates (parameterized patterns like `file:///{path}` that clients expand to concrete URIs)
- `resources/subscribe` — subscribe to change notifications for a specific resource URI
- `resources/unsubscribe` — cancel a subscription

Clients receive a `notifications/resources/updated` notification when subscribed resource content changes, and a `notifications/resources/list_changed` notification when the available resource list itself changes.

**Concrete example:** A coding assistant server exposes the current open file as `file:///current-file`. The host app calls `resources/read` when the user opens the file; the model sees the content as context, never having "requested" it.

### Prompts (user-controlled)

Prompts are reusable, parameterized message templates that a server exposes. The user explicitly selects one — typically via a slash command or UI affordance in the host client. The model does not invoke prompts autonomously.

**JSON-RPC methods:**

- `prompts/list` — enumerate available prompt templates (name, description, argument list)
- `prompts/get` — retrieve a rendered prompt for a given name and argument values

A `notifications/prompts/list_changed` notification signals when the prompt catalog updates.

**Concrete example:** A code-review server exposes a `review_pr` prompt with a `pr_url` argument. The user types `/review_pr pr_url=https://github.com/…`; the client calls `prompts/get` and inserts the rendered messages into the conversation.

### Tools (model-controlled, brief)

Tools are executable actions the model invokes autonomously based on context. Listed via `tools/list`, called via `tools/call`. This is the most-used primitive; see /resources/mcp-vs-a2a for the full protocol context.

## Client-side primitives

These primitives invert the direction: the server calls into the client rather than the client calling the server.

### Sampling (server calls client for an LLM completion)

Sampling lets a server request an LLM completion from the host client on its behalf. The server sends a `sampling/createMessage` request containing a message list, an optional system prompt, and optional model preferences (cost/speed/intelligence hints). The client — subject to user approval — forwards the request to its connected LLM and returns the result.

**JSON-RPC method:** `sampling/createMessage` (server → client)

The human-in-the-loop design is deliberate. The spec requires clients to let users review and modify sampling requests before sending them to the model, and to review model responses before returning them to the server. This ensures the server cannot silently use the model to exfiltrate context or generate content the user has not sanctioned.

Note: Sampling is marked deprecated in the draft 2026-07-28 spec revision (SEP-2577). Per the MCP feature lifecycle policy it remains in the 2025-11-25 stable spec and will not be eligible for removal for at least twelve months after that draft ships.

**Concrete example:** A database MCP server receives a `resources/read` call for a large schema blob. Rather than hard-coding a summary, it calls `sampling/createMessage` asking the LLM to condense the schema into a two-sentence description, then returns that summary as the resource content.

### Roots (client declares filesystem/URI boundaries)

Roots are URIs (commonly `file://` paths) that the client declares to tell the server which parts of the filesystem or resource space it is permitted to access. This is a coordination mechanism, not a hard security enforcement: the spec requires servers to respect root boundaries, but cannot enforce it because servers run code outside the client's control.

**JSON-RPC methods:**

- `roots/list` — server requests the current list of roots from the client
- `notifications/roots/list_changed` — client notifies the server when the root list changes

Roots update dynamically: when a user opens a new project folder, the client sends `notifications/roots/list_changed` and the server can call `roots/list` to refresh its scope — no server restart required.

**Concrete example:** A filesystem MCP server starts with roots `[file:///projects/alpha]`. The user opens a second project; the client sends `notifications/roots/list_changed`; the server calls `roots/list` and now has `[file:///projects/alpha, file:///projects/beta]`.

Note: like Sampling, Roots is marked deprecated in the draft 2026-07-28 spec revision (SEP-2577, which deprecates roots, sampling, and logging). It remains in the 2025-11-25 stable spec and stays usable for at least twelve months after that draft ships.

### Elicitation (server requests structured user input)

Elicitation was introduced in the **2025-06-18** spec revision. It lets a server request additional structured input from the user mid-session, without the server needing to implement its own UI.

**JSON-RPC method:** `elicitation/create` (server → client)

The request carries a human-readable `message` and an optional `requestedSchema` — a restricted JSON Schema subset (flat object with primitive-type properties only; no nested objects or arrays). The client renders the schema as a form, the user fills it in (or declines), and the client returns one of three actions:

- `accept` — user submitted the form; response includes the filled content object
- `reject` — user explicitly declined to provide the information
- `cancel` — user dismissed the dialog without taking action

Servers are explicitly prohibited from using elicitation to request sensitive data (passwords, tokens, credentials). The user can always reject an elicitation request.

**Concrete example:** A CI/CD MCP server is about to deploy to production. It calls `elicitation/create` with `message: "Confirm deployment target"` and a schema requiring `{ env: string, confirm: boolean }`. The user fills in the form; the server proceeds only on `action: accept` with `confirm: true`.

## Resources vs Tools: the key design question

The most common MCP server design question is whether to expose something as a Resource or a Tool. The spec's control-model distinction gives a clean answer:

| Use a Resource when... | Use a Tool when... |
|---|---|
| The data is read-only context the app injects | The model needs to trigger an action or side effect |
| The app (not the model) decides what to include | The model decides whether and when to call |
| The content is stable or subscribable | The content depends on model-chosen arguments |
| Example: open file, DB row, live feed | Example: search API, code executor, write operation |

A file viewer is a Resource. A file writer is a Tool. A search endpoint that takes a query the model chooses is a Tool. A configuration blob the app always injects is a Resource.

## Verified sources

- MCP schema 2025-11-25 (method names verified directly): https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/schema/2025-11-25/schema.ts
- MCP elicitation spec 2025-06-18 (introduction revision confirmed): https://modelcontextprotocol.io/specification/2025-06-18/client/elicitation
- IBM mcp-context-forge issue #234 (elicitation/create confirmed in 2025-06-18): https://github.com/IBM/mcp-context-forge/issues/234
- MCP resources spec (resources/list, resources/read, resources/subscribe confirmed): https://spec.modelcontextprotocol.io/specification/server/resources/
- Speakeasy MCP roots reference (roots/list, notifications/roots/list_changed): https://www.speakeasy.com/mcp/core-concepts/roots
- MCP sampling spec (sampling/createMessage): https://spec.modelcontextprotocol.io/specification/2025-03-26/client/sampling/
- MCP vs A2A protocol comparison: /resources/mcp-vs-a2a
- MCP server discovery guide: /resources/mcp-server-discovery


Multimodal Agents: Vision, Documents, and Screens

How agents perceive and reason over images: VLM mechanics, image-input APIs across major providers, open-weight VLM families, grounding/pointing, failure modes, and practical guidance for agent builders.

Multimodal agents extend the standard text-in / text-out loop with image input. A vision-language model (VLM) accepts one or more images alongside the text prompt and reasons over both jointly. This unlocks agent use cases that are impossible with text alone: reading UI screenshots, interpreting charts and diagrams, parsing scanned documents, and answering visual questions about photos or camera feeds.

This guide covers how image input works at the API level, what VLMs can and cannot do, the verified open-weight families, and practical guidance for building reliable pipelines. Computer-use (acting on UIs via screenshots) is covered at /resources/computer-use-browser-automation; document OCR pipelines are at /resources/document-extraction-for-agents.

## What "multimodal" means for agents

A VLM fuses a vision encoder (typically a ViT-based model that converts an image to patch embeddings) with a language model backbone. The image embeddings are projected into the same token space as text, so the LLM sees a mixed sequence of vision tokens and text tokens and attends over both simultaneously.

Common agent use cases for image input:

- **UI / screenshot reading** — parsing a rendered screen to identify elements, extract text, or determine what to click next. See /resources/computer-use-browser-automation.
- **Chart and diagram interpretation** — reading bar charts, line graphs, flowcharts, or architectural diagrams embedded in reports.
- **Document and PDF parsing** — passing page images to a VLM instead of (or alongside) an OCR step. See /resources/document-extraction-for-agents.
- **Photo and scene understanding** — answering questions about real-world images: "Is the product label intact?", "How many items are on the shelf?"
- **Visual QA in agentic loops** — using an image as evidence that a prior action succeeded (e.g., a confirmation screenshot after form submission).

## How image input works at the API level

### OpenAI (GPT-4o legacy; GPT-4.1 and GPT-5.x current)

Images are passed inside the `messages` array as content blocks of type `image_url`. Two delivery methods: a fully qualified HTTPS URL, or a base64-encoded data URL (`data:image/png;base64,...`). An optional `detail` parameter accepts `"low"`, `"high"`, or `"auto"`; low detail processes a fixed low-resolution version of the image (cheaper). On the legacy GPT-4o model, high detail tiles the image into 512×512 pixel segments and processes each tile separately, paying more tokens proportional to image area. As of July 2026, per multiple agreeing secondary sources (not independently primary-fetched this cycle), GPT-4.1 and the current GPT-5.x family instead use a patch-based scheme — the image is covered in 32×32-pixel patches up to a model- and detail-level-specific patch budget — rather than 512×512 tiling; check the specific model's own docs before building a cost model. Images can also be referenced via the Files API using a file ID. Supported formats: JPEG, PNG, GIF, WebP. Source: platform.openai.com/docs/guides/images-vision.

### Anthropic Claude (3/4 families)

Images are passed as `image` content blocks inside the `messages` array. Three delivery methods: base64-encoded image data with an explicit `media_type` (`image/png`, `image/jpeg`, `image/gif`, `image/webp`), a URL reference to a hosted image, or a file ID from the Files API. The API accepts up to 100 images per request for models with a 200k-token context window, or up to 600 images per request for other models (up to 20 per message on claude.ai); a stricter per-image limit (resize to 2000 px or under on the long edge) applies once a single request carries more than 20 images. Images larger than 8000×8000 px are rejected outright. Claude tokenizes images in 28×28-pixel patches and resizes/pads large images to its native resolution before processing; newer models (e.g. Opus 4.8, Sonnet 5) support a higher-resolution tier than older ones. Source: docs.anthropic.com/en/docs/build-with-claude/vision.

### Google Gemini

Images are passed as inline data parts (base64 with MIME type) or via the File API (recommended once total request size exceeds 100 MB, or for reusing images across requests — as of a January 2026 Google update per multiple agreeing secondary sources including a Google staff account, not independently primary-fetched this cycle, which raised the inline-payload ceiling from a prior 20 MB). Gemini models were designed multimodal from the ground up; all current Gemini models accept image input alongside text in a unified context. The File API also supports video, audio, and PDF inputs. Source: ai.google.dev/gemini-api/docs/image-understanding.

### Token-cost impact

Image tokens are significantly more expensive than text tokens per pixel of information. Qualitatively: larger images, higher detail settings, and tile-based processing each increase the token count (and cost) proportionally. Downscaling images to the minimum resolution needed for the task is the primary cost lever. Exact token formulas differ by provider and model version — check provider pricing pages for current figures before building cost models.

## Capabilities and limits

### What VLMs can do well

- **Native image understanding** — reading text in images (OCR-class), describing scenes, answering "what is in this image?" questions.
- **Chart and diagram comprehension** — extracting values from bar charts, understanding flowcharts, describing architectural diagrams.
- **Multi-image reasoning** — comparing two images, spotting differences, tracking changes across a sequence of screenshots.
- **Grounding and pointing** — some models can return bounding-box coordinates or pixel points in response to "where is X in this image?". Molmo 2 (AI2) and Qwen3-VL both support returning spatial coordinates. This enables downstream click-targeting without a separate detection model.
- **Video frames** — most major provider models accept video frames or short clips; check current documentation for frame-count and duration limits.

### Known failure modes

| Failure mode | Description |
|---|---|
| Small or low-contrast text | Dense fine-print, watermarks, or text on complex backgrounds frequently mis-read. Combine OCR + VLM for precision-critical fields. |
| Dense tables | Complex spanning cells and merged headers are prone to misalignment or omission. Verify extracted table content against source. |
| Spatial reasoning | Counting objects, estimating distances, or reasoning about exact pixel positions is unreliable without explicit grounding support. |
| Hallucinated visual details | VLMs sometimes describe objects or text that are not present. Treat image-derived facts as unverified until cross-checked. |
| Prompt injection via images | Adversarial instructions embedded as text within an image enter the model's attention pathway and may override system-prompt constraints. See /resources/agentic-security-checklist. |

## Verified open-weight VLM families

All families below are confirmed to exist and have open weights on Hugging Face. No benchmark rankings are published here — rankings shift frequently and depend heavily on task and resolution.

- **Qwen3-VL** (Alibaba/QwenLM, Apache 2.0) — 2B/4B/8B/32B dense plus MoE variants (30B-A3B, 235B-A22B). Supports native image and video input; returns spatial coordinates for grounding tasks. Hugging Face: huggingface.co/Qwen/Qwen3-VL-8B-Instruct (and other sizes).
- **Llama 3.2 Vision** (Meta, Llama Community License) — 11B and 90B vision-language models released September 2024, built on the Llama 3.1 backbone. Hugging Face: huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct.
- **Molmo 2** (Allen Institute for AI / AI2, permissive open license) — 4B and 8B models (Qwen 3 backbone) plus a 7B (OLMo backbone). Specializes in precise spatial grounding and pointing: the model can return pixel-level coordinates indicating where an object or element is located in an image. Video and multi-image understanding built in. Released December 2025. Hugging Face: huggingface.co/allenai/Molmo2-8B.
- **Pixtral** (Mistral AI, model weights available) — Pixtral Large is a 124B open-weight multimodal model built on Mistral Large 2. Strong image understanding; weights released under Mistral's license. Mistral AI: mistral.ai/news/pixtral-large/ (note: listed as deprecated on Mistral's site as of mid-2026 — verify before using in new projects).
- **InternVL3** (Shanghai AI Lab / OpenGVLab, MIT license) — successor to InternVL 2 and 2.5; sizes from 1B to 78B. Supports multi-image input, document understanding, and tool use. Hugging Face: huggingface.co/OpenGVLab/InternVL3-78B.
- **Gemma 3 / Gemma 4** (Google) — Gemma 3 (1B/4B/12B/27B, multimodal with image input, released March 2025) under the custom Gemma Terms of Use; Gemma 4 (multimodal, released April 2026) is the first Gemma released under true Apache 2.0 (see /resources/open-weight-models-for-agents). Available on Hugging Face under the Google organization.

## Practical guidance for agent builders

**Control image size and cost.** Downscale images to the minimum resolution that preserves the information your task needs before sending them to the API. A screenshot for UI parsing rarely needs more than 1280px on the long edge. Unnecessary resolution multiplies token cost with no quality gain.

**Prefer structured prompts for extraction tasks.** Instead of "describe this image," use prompts like "List every line item, quantity, and unit price visible in this invoice table." Specificity reduces hallucinated detail and improves parseable output.

**Combine OCR + VLM for precision-critical fields.** For financial, medical, or legal documents where small text or dense tables matter, run a dedicated OCR pass (see /resources/document-extraction-for-agents) and feed the OCR text alongside the page image. The VLM then reasons over both, using the OCR text as a ground-truth anchor while the image provides layout context.

**Validate extracted facts.** Treat all image-derived values as unverified until cross-checked against a schema or a second pass. A VLM hallucinating a dollar amount or a part number is a real production failure mode.

**Use structured outputs for downstream consumption.** Pair VLM calls with strict JSON Schema constraints (see /resources/reliable-tool-calling) so that extracted fields arrive in a parseable, typed form rather than free text.

**Treat image content as untrusted.** Text visible in an image — from a rendered web page, a user-uploaded screenshot, or a scanned document — can contain adversarial prompt-injection payloads. The model processes that text as part of its context and may act on embedded instructions. Apply the same untrusted-input mitigations as for text from the web. See /resources/agentic-security-checklist (section 1, prompt injection).

**Cross-links:** /resources/computer-use-browser-automation (acting on UI screenshots) · /resources/document-extraction-for-agents (OCR + document parsing pipelines) · /resources/reliable-tool-calling (structured output from VLM extraction) · /resources/agentic-security-checklist (prompt injection via images)

## Verified sources

- OpenAI images and vision docs: https://platform.openai.com/docs/guides/images-vision
- Anthropic Claude vision docs: https://docs.anthropic.com/en/docs/build-with-claude/vision
- Google Gemini image understanding docs: https://ai.google.dev/gemini-api/docs/image-understanding
- Qwen3-VL-8B-Instruct (Hugging Face): https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct
- Llama 3.2 Vision (Hugging Face, Meta): https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct
- Molmo 2 announcement (AI2): https://allenai.org/blog/molmo2
- Molmo2-8B (Hugging Face, AI2): https://huggingface.co/allenai/Molmo2-8B
- Pixtral Large announcement (Mistral AI): https://mistral.ai/news/pixtral-large/
- InternVL3-78B (Hugging Face, OpenGVLab): https://huggingface.co/OpenGVLab/InternVL3-78B
- Gemma 3 multimodal guide (Roboflow): https://blog.roboflow.com/gemma-3/
- Image-based prompt injection research (CSA, 2026): https://labs.cloudsecurityalliance.org/research/csa-research-note-image-prompt-injection-multimodal-llm-2026/
- What does it cost to process an image with a vision model? (Roboflow, on GPT-4.1/GPT-5.x patch-based tokenization): https://blog.roboflow.com/image-token-cost-vlm/
- Increased file size limits and expanded inputs support in Gemini API (Google): https://blog.google/innovation-and-ai/technology/developers-tools/gemini-api-new-file-limits/


Agent Identity and Authentication

How autonomous agents prove who they are and get authorized to act: workload identity vs. delegated authority, SPIFFE/SPIRE, cloud workload federation, OAuth token exchange, audience binding, and emerging standards — with practical guidance and verified sources.

An autonomous agent faces a credential problem that neither classic human login nor static service accounts solve cleanly. The agent must prove two distinct things: (1) what it *is* — its own workload identity — and (2) that it has *permission to act on behalf of* a human or organization. Conflating these two layers is the root cause of most agent auth mistakes.

## Key facts

- Agents need to prove two separate things — a cryptographic identity for the running workload itself, and a distinct authorization grant to act on a human's or organization's behalf — and mixing up these two layers causes most agent-auth failures.
- Static, long-lived keys shared across instances are a poor fit for workload identity because they can't be pinned to one running process, don't auto-rotate, and give indefinite access if leaked — the fix is short-lived, cryptographically attested credentials such as SPIFFE/SPIRE's SVIDs, issued only after the SPIRE server checks platform-level launch evidence.
- The same idea extends across clouds: a workload can trade the OIDC token its home platform issued it for temporary credentials in a different cloud through a security token service, so no static secret ever has to be shared between the two.
- For delegated authority, scoped OAuth grants are the standard, and letting an agent simply hold and forward the user's own token is an anti-pattern — it invites overbroad access, no distinct audit trail, and confused-deputy replay against services the token wasn't meant for.
- Two IETF mechanisms close that gap: token exchange (RFC 8693) lets an agent swap a held token for a narrower one for a sub-task, and resource-indicator audience binding (RFC 8707) ties a token's audience claim to one specific resource server so it can't be replayed elsewhere.
- Practical hardening layers per-agent unique credentials, short token lifetimes, mandatory audience binding and token exchange over passthrough, built-in rotation/revocation, full per-identity audit logging, and human sign-off before any privilege escalation.

## The two-layer model

**Layer 1 — Workload / agent identity (what the agent is)**
The agent itself needs a machine identity it can cryptographically prove. This is analogous to a TLS server certificate — but issued dynamically to a running workload rather than a static host. Long-lived API keys shared across agent instances are dangerous: they cannot be tied to a specific running process, they do not rotate automatically, and a leaked key grants permanent access.

**Layer 2 — Delegated authority (permission to act on behalf of a user)**
When an agent takes actions in systems a human owns — reading email, calling an API with the user's permissions, submitting forms — it needs an authorization grant from that human. Holding the user's own credentials is the wrong pattern: it grants unlimited access with no scope boundary and no audit trail separating human actions from agent actions.

## Workload identity: short-lived credentials and SPIFFE/SPIRE

The right approach for workload identity is cryptographically attested, short-lived credentials issued to a specific running workload — not a human typing a password or a static token in an environment variable.

**SPIFFE/SPIRE** (Secure Production Identity Framework for Everyone / SPIFFE Runtime Environment) is the most widely adopted open standard for workload identity. SPIFFE and SPIRE graduated from CNCF incubation in September 2022. The framework issues each workload a SPIFFE Verifiable Identity Document (SVID) in one of two established forms, plus an emerging third:

- **X.509-SVID** — an X.509 certificate embedding the workload's SPIFFE ID in the Subject Alternative Name field as a URI (`spiffe://trust-domain/path`). This certificate is used directly for mutual TLS (mTLS), enabling two workloads to authenticate each other cryptographically before any application data flows.
- **JWT-SVID** — a signed JWT carrying the SPIFFE ID, presented as a plain bearer token in an Authorization header — suitable for HTTP/REST contexts.
- **WIT-SVID** — a newer SPIFFE profile of the IETF WIMSE working group's own Workload Identity Token (WIT) format, added to the SPIFFE spec at "Incubating" stability (below "Stable") as of mid-2026. Unlike JWT-SVID, it binds the token to the workload's key-pair via a `cnf` (confirmation) claim, so it must be presented with proof of possession rather than as a bare bearer token — a complement to JWT-SVID, not a replacement, and the SPIFFE spec's own bridge to WIMSE-compatible workload identity.

SVIDs are short-lived and automatically rotated by the SPIRE agent — no human intervention needed. The SPIRE server attests the identity of a workload at launch time using platform-level evidence (Kubernetes service account token, AWS instance identity document, GCP metadata service token, etc.) and only then issues an SVID. This is the key distinction from a static API key: the credential is tied to a provably running workload, not to a secret that can be copied and reused.

See /resources/agentic-security-checklist (section 5 — secrets and credential management) for why long-lived shared API keys are dangerous for agents.

**Cloud workload identity federation** — all three major cloud providers support OIDC-based workload identity federation: a workload running on one platform (e.g. AWS EC2, a GitHub Actions runner, or a Kubernetes pod) exchanges its platform-issued OIDC token for short-lived credentials in a target cloud (e.g. GCP IAM, Azure Entra ID, AWS IAM). No static cross-cloud secret is required; the exchange is mediated by a Security Token Service. GCP's Workload Identity Federation, AWS IAM Roles Anywhere, and Azure Managed Identities are the canonical implementations. See the GCP documentation at cloud.google.com/iam/docs/workload-identity-federation.

## Delegated authority: OAuth and token exchange

**OAuth 2.0 authorization code + scopes** is the standard protocol for an agent to obtain a bounded grant from a user. The user authorizes specific scopes; the agent receives a token that can only perform those scoped operations. Least-privilege scoping is essential: request only the scopes the current task requires, not a broad grant for all possible future tasks.

**The agent-holds-user-token anti-pattern** — storing the user's own access token (or refresh token) in the agent is risky: the token may have broad scopes beyond what the task needs, the agent has no distinct audit identity, and token leakage grants attacker-level access to the user's account.

**RFC 8693 — OAuth 2.0 Token Exchange** (IETF, January 2020) defines a protocol for an agent to present a token it already holds and receive a *different*, narrower token in return. This is the canonical mechanism for an agent to obtain a downstream token scoped to a specific sub-task or service, without holding the user's original broad token. The grant type is `urn:ietf:params:oauth:grant-type:token-exchange`.

**Confused-deputy and token-passthrough risk** — an agent that receives a user token from an orchestrator and forwards it directly to a downstream service creates a confused-deputy vulnerability: the downstream service cannot distinguish a legitimate orchestrator request from a forged one, and the token audience is not validated. RFC 8693 token exchange, combined with RFC 8707 resource indicators (audience binding), closes this gap. See /resources/mcp-server-authentication for how MCP explicitly forbids token passthrough.

**RFC 8707 — Resource Indicators for OAuth 2.0** (IETF, February 2020) adds a `resource` parameter to OAuth requests so the client declares which resource server the token is for. The authorization server then issues a token whose audience (`aud` claim) is bound to that specific resource, preventing a token issued for Service A from being replayed against Service B.

## Emerging agent-auth standards (as of June 2026)

Active standardization work addresses the agent-specific gaps in existing protocols:

**IETF WIMSE** — the Workload Identity in Multi-System Environments working group was chartered in March 2024 at IETF 119. It is producing architecture documents and token-exchange profiles for workloads that need to authenticate and exchange credentials across heterogeneous platforms. WIMSE explicitly addresses the case where an agent receives a workload identity token and must present it to a downstream system that uses a different identity scheme.

**OpenID Foundation — Identity Management for Agentic AI** — the OpenID Foundation's AI Identity Management Community Group published a whitepaper in October 2025 identifying two authentication layers that must work in tandem: the agent software itself authenticated as a trusted client, and the human delegating specific permissions. The Foundation has also attracted individual Internet-Drafts (e.g. draft-sharif-openid-agent-identity-00, March 2026) proposing agent-specific claims for OpenID Connect ID tokens — ownership, trust posture, and authorised capabilities. These are individual drafts with no formal IETF working-group standing as of June 2026, but they signal the direction the identity community is moving.

## Practical guidance

- **Give each agent instance its own identity.** No shared keys, no shared OAuth client IDs across agent instances. Per-agent registration means a compromised credential affects one agent, not all of them.
- **Issue short-lived, scoped tokens.** Tokens should expire in minutes to hours, not days. Use refresh tokens or re-attest via SPIRE to get fresh credentials when needed.
- **Audience-bind every token (RFC 8707).** Include the `resource` parameter in OAuth requests so tokens cannot be replayed against unintended services.
- **Use token exchange (RFC 8693) rather than passing tokens through.** When an agent delegates to a downstream service, exchange for a narrower token scoped to that service.
- **Rotate and revoke.** Build revocation into your agent lifecycle: when a task ends, revoke the tokens; when an agent is decommissioned, rotate secrets and expire credentials.
- **Full audit log of which identity did what.** Every action taken by an agent should be attributed to that agent's unique identity in an append-only audit log. See /resources/agent-observability for the broader tracing and logging pattern.
- **Require human approval for high-privilege grants.** Agents should not be able to self-elevate privilege. Any grant of admin-level or broad-scope access should require an explicit human approval step. See /resources/agent-guardrails.

## Cross-links

- OAuth for MCP servers specifically: /resources/mcp-server-authentication
- Secrets and credential management in agent pipelines: /resources/agentic-security-checklist
- Audit logging and tracing of agent actions: /resources/agent-observability
- Human approval gates for high-stakes actions: /resources/agent-guardrails

## Verified sources

- SPIFFE/SPIRE CNCF graduation announcement (September 2022): https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/
- SPIFFE overview and SVID specification: https://spiffe.io/docs/latest/spiffe-about/overview/
- SPIFFE WIT-SVID specification (spiffe/spiffe repo, standards/WIT-SVID.md): https://github.com/spiffe/spiffe/blob/main/standards/WIT-SVID.md
- RFC 8693 — OAuth 2.0 Token Exchange (IETF, January 2020): https://datatracker.ietf.org/doc/html/rfc8693
- RFC 8707 — Resource Indicators for OAuth 2.0 (IETF, February 2020): https://datatracker.ietf.org/doc/html/rfc8707
- IETF WIMSE working group (chartered March 2024): https://datatracker.ietf.org/group/wimse/history/
- WIMSE architecture draft: https://www.ietf.org/archive/id/draft-ietf-wimse-arch-01.html
- OpenID Foundation — Identity Management for Agentic AI whitepaper (October 2025): https://openid.net/wp-content/uploads/2025/10/Identity-Management-for-Agentic-AI.pdf
- OpenID Foundation — new whitepaper on AI agent identity challenges: https://openid.net/new-whitepaper-tackles-ai-agent-identity-challenges/
- GCP Workload Identity Federation documentation: https://cloud.google.com/iam/docs/workload-identity-federation


Agent Delegation Chains: Credential Propagation in Multi-Agent Systems

How credential authority flows when one agent spawns another — the multi-hop delegation problem, RFC 8693 token exchange, the act/may_act claims, audience binding across hops, and the IETF drafts standardizing verifiable actor chains in 2026.

When an orchestrator agent delegates a sub-task to a child agent, which in turn calls a tool API, a multi-hop credential chain exists. Each hop must answer the same question: "who authorized this, and is that authorization still valid?" Standard OAuth 2.0 and workload identity solve the single-agent case (see /resources/agent-identity-authentication); multi-hop delegation needs additional mechanisms that are only now being standardized.

## Key facts

- Delegation across several hops — say a top-level coordinator handing work to a subordinate agent that then calls an external tool — forces every link to independently re-confirm who granted permission and whether that grant is still active, a question ordinary single-agent OAuth and workload identity don't fully answer.
- RFC 8693's token exchange supports delegation through two claims: a nestable `act` claim that appends each successive actor to build a chain, and a `may_act` claim that lets the token issuer pre-approve who's allowed to act next.
- RFC 8693 itself doesn't specify how to verify or bound that nested chain cryptographically — each exchange only checks the immediately preceding hop, so nothing stops delegation depth from growing unchecked or the recorded actor history from being purely informational.
- Binding each hop's token audience to only the next service in line (RFC 8707) stops a token grabbed mid-chain from working against some other downstream service the same authorization server trusts — the confused-deputy flaw, replayed across hops instead of a single call.
- Two still-unratified IETF drafts push this further: one defines verifiable, tamper-resistant actor-chain profiles that block an attacker from splicing themselves into the chain, and the other adds explicit per-agent user consent parameters to the standard authorization-code flow.
- Recommended guardrails for builders include capping how many hops a delegation may traverse, keeping each hop's token lifetime to minutes, recording the complete actor chain wherever the resource server logs requests, and always minting a fresh downstream token rather than forwarding one received from upstream.

## The multi-hop delegation problem

RFC 8693 (OAuth 2.0 Token Exchange, IETF, January 2020) is the canonical mechanism for one service to exchange a token it holds for a narrower one. It defines the grant type `urn:ietf:params:oauth:grant-type:token-exchange` and two JWT claims relevant to delegation:

- **`act` claim** — identifies the party currently acting on behalf of the token subject. The claim is nestable: each hop appends itself, creating an actor chain. Example: orchestrator agent → sub-agent → tool API produces a nested `act` chain.
- **`may_act` claim** — the subject token names a party explicitly authorized to act on the subject's behalf. It is the forward-authorization mechanism: the issuer pre-approves which actor may operate next in the chain.

The gap RFC 8693 leaves open: it does not define how an authorization server validates or enforces delegation-path *continuity* across successive exchanges. Each exchange is point-to-point; prior actors in the nested `act` claim are informational only, with no standard way to verify the full chain cryptographically or to cap delegation depth.

## Audience binding across hops

RFC 8707 (Resource Indicators for OAuth 2.0, IETF, February 2020) lets each token request name the specific downstream service via the `resource` parameter, binding an audience (`aud`) to the issued token. In a delegation chain, each hop should request a token whose audience is the *next* service only. A token minted for service B must not be accepted by service C — without this, a token captured at hop 2 can be replayed against any service that trusts the same authorization server: the confused-deputy attack applied to chains.

## Emerging IETF work (individual Internet-Drafts as of June 2026)

**draft-mw-spice-actor-chain** (IETF SPICE space; rev -05, April 2026) — "Cryptographically Verifiable Actor Chains for OAuth 2.0 Token Exchange." Defines six actor-chain profiles separating disclosure level (Full / Subset / Actor-Only) from trust model (Declared vs Verified). Verified profiles require sender-constrained tokens and a signature over the chain, preventing chain-splicing where an attacker inserts themselves between legitimate hops. Preserves the existing meaning of `sub`, `act`, and `may_act`; supports same-domain and cross-domain delegation. This work has since been resubmitted into the IETF OAuth working group's naming space as **draft-mw-oauth-actor-chain-00**, seeking formal WG adoption under the same title and profile structure — treat the OAuth-WG draft, not the original SPICE-space document, as the one to track going forward.

**draft-oauth-ai-agents-on-behalf-of-user** (individual, rev -02, August 2025) — extends the OAuth 2.0 authorization-code flow with two parameters: `requested_actor` in the authorization request (names the specific agent being delegated to, for explicit user consent) and `actor_token` in the token request (the agent authenticates itself when exchanging the code). The resulting access token carries an `act` claim naming the agent, separating user identity from agent identity in the audit record.

**Note:** both are Internet-Drafts with no IETF working-group RFC standing as of June 2026. Treat them as directional signals, not stable specifications — verify the current revision on the IETF Datatracker before building to them.

## Standards bodies are converging on this

NIST's National Cybersecurity Center of Excellence (NCCoE) published the concept paper "Accelerating the Adoption of Software and AI Agent Identity and Authorization" on 5 February 2026 (public comment closed 2 April 2026), scoping a potential practical-guide project. It frames the problem around identifying AI agents distinctly from human users, authorizing their actions, delegating access, and auditing/non-repudiation of agent actions — the same control areas this resource covers. It signals that agent delegation is moving from ad-hoc practice toward documented reference architectures.

## Practical patterns for multi-agent builders

| Pattern | Mechanism | Key property |
|---|---|---|
| Per-hop token exchange | RFC 8693 `grant-type:token-exchange` | Each hop holds the narrowest token it needs |
| Audience binding | RFC 8707 `resource` parameter | A token replayed to the wrong service is rejected |
| Chain auditability | Nested `act` claim | Full delegation path recorded in every token |
| Forward authorization | `may_act` claim in the subject token | Issuer pre-approves the next actor |
| Cryptographic chain integrity | draft-mw-spice-actor-chain Verified profiles | Prevents chain-splicing attacks |
| Agent-specific consent | draft-oauth-ai-agents-on-behalf-of-user `requested_actor` | User explicitly authorizes a named agent |

- **Enforce a maximum delegation depth.** Decide at authorization-server configuration time how many hops are permitted and reject exchanges that would exceed it. No current RFC mandates this; it is an application-layer policy.
- **Use short-lived tokens at every hop.** A 5–15 minute lifetime per step bounds the blast radius to a single hop if a token is exfiltrated.
- **Log the full `act` chain at the resource server.** The nested `act` claim is the only audit artifact linking the original user delegation to every downstream agent action.
- **Do not pass tokens straight through.** An MCP server receiving a Bearer token from an orchestrator should obtain its own downstream token from the authorization server rather than forwarding the received one. See /resources/mcp-server-authentication.

## Related ChangeGamer resources

- Single-agent workload identity and auth fundamentals: /resources/agent-identity-authentication
- MCP server OAuth flows (audience binding, token-passthrough prohibition): /resources/mcp-server-authentication
- Multi-agent orchestration topologies that create these chains: /resources/multi-agent-orchestration-patterns
- Credential management and scopes in the broader checklist: /resources/agentic-security-checklist

## Verified sources

- RFC 8693 — OAuth 2.0 Token Exchange (IETF, January 2020): https://datatracker.ietf.org/doc/html/rfc8693
- RFC 8707 — Resource Indicators for OAuth 2.0 (IETF, February 2020): https://datatracker.ietf.org/doc/html/rfc8707
- draft-mw-spice-actor-chain (Cryptographically Verifiable Actor Chains): https://datatracker.ietf.org/doc/draft-mw-spice-actor-chain/
- draft-oauth-ai-agents-on-behalf-of-user (On-Behalf-Of User Authorization for AI Agents): https://datatracker.ietf.org/doc/draft-oauth-ai-agents-on-behalf-of-user/
- NIST NCCoE concept paper — Software and AI Agent Identity and Authorization (Feb 2026): https://www.nccoe.nist.gov/news-insights/new-concept-paper-identity-and-authority-software-agents
- NIST CSRC publication record (5 Feb 2026): https://csrc.nist.gov/pubs/other/2026/02/05/accelerating-the-adoption-of-software-and-ai-agent/ipd
- draft-mw-oauth-actor-chain-00 — OAuth WG resubmission of the actor-chain work (WebSearch-fallback, 2 sources): https://datatracker.ietf.org/doc/draft-mw-oauth-actor-chain/