AI Gateways and LLM Routing

What an AI gateway is, routing strategies (failover, cost-cascade, latency, capability), the tooling landscape, the OpenAI-compatible API convention, and tradeoffs.

An **AI gateway** (also called an LLM proxy or LLM router) is a single reverse-proxy that sits in front of multiple model providers and exposes one unified endpoint to callers. Instead of coding provider-specific clients for OpenAI, Anthropic, Google, Bedrock, and others, every agent sends requests to the gateway; the gateway routes, retries, caches, rate-limits, and logs on behalf of the caller.

For production agents, a gateway solves five categories of problems simultaneously: provider reliability (failover and retries), cost control (budget caps, model cascades), performance (caching, load balancing), security (key management, PII redaction), and observability (centralized logging of every request — see /resources/agent-observability).

## Key facts

- An AI gateway is a single reverse-proxy standing between an agent and several different model providers at once, giving agents one unified endpoint instead of provider-specific client code.
- A gateway tackles five problem areas for production agents at once: provider reliability, cost control, performance, security, and observability.
- Routing strategies span failover, latency-based, cost-based/model-cascade, and capability-based approaches, each suited to a different tradeoff.
- Most gateways speak the de-facto OpenAI-compatible chat-completions request/response shape, which is what lets an agent swap its upstream provider with no code change.
- The tooling landscape splits between self-hosted, open-source options (LiteLLM, Portkey, Kong) and hosted SaaS aggregators (OpenRouter, Cloudflare AI Gateway, Helicone).
- Tradeoffs include the added network hop's latency, a new single point of failure, and data-privacy exposure since every prompt and response passes through the gateway operator.
- Self-hosting trades operational effort for control, while SaaS gateways stay lightweight to run but create a dependency on the vendor.

## Core gateway capabilities

- **Unified endpoint** — one `POST /v1/chat/completions`-compatible URL in front of many providers.
- **Provider failover** — if provider A returns 5xx or times out, retry on provider B automatically.
- **Load balancing** — distribute traffic across multiple deployments of the same model.
- **Caching** — exact-match response caching for repeated prompts; semantic caching for near-duplicate queries using embedding similarity (see /resources/agent-memory-context).
- **Rate limiting and budget controls** — per-key, per-team, or per-model spend caps and request-per-minute limits enforced before upstream charges accrue.
- **Key management** — virtual keys issued to callers; the gateway holds the real provider keys.
- **Observability** — every request logged with latency, cost, tokens, and model used (see /resources/agent-observability).

## Routing strategies

**Provider failover** — primary provider first; on error (5xx, timeout) automatically retry on one or more fallback providers. Protects uptime without code changes in the agent.

**Latency-based routing** — measure provider response time and weight traffic toward the fastest responder. Useful when several providers offer the same model (e.g., via OpenRouter).

**Cost-based / model cascade** — route the request to the cheapest model capable of handling it. A classifier or confidence score determines whether to escalate to a more capable (and expensive) model. Also called "mixture-of-agents" or "model cascade." RouteLLM (see below) implements this as a standalone router.

**Capability routing** — send image inputs to a vision-capable model, audio to a speech-capable model, and code to a code-specialized model, based on detected input type or an explicit hint in the request.

## The OpenAI-compatible API convention

Most gateways and aggregators expose the `POST /v1/chat/completions` endpoint shape (request: `model`, `messages`, `temperature`; response: `choices[].message`). This is a de-facto industry standard, not a ratified spec — it originated with OpenAI and was copied by most providers. The newer `POST /v1/responses` shape (OpenAI Responses API, released March 2025) is increasingly supported but adoption is not yet universal. Because clients target this shape, swapping the gateway's upstream provider requires no code change on the agent side.

## Tooling landscape

**LiteLLM** — open-source proxy (MIT license; the `enterprise/` directory ships under a separate commercial license) that added a Rust core alongside its Python SDK; its own GitHub tagline now reads "the fastest, litest AI Gateway... Call 100+ LLM APIs" (as of July 2026), a headline-number and architecture shift from the project's earlier "140+ providers / 2,600+ models" framing — the underlying provider table still lists well over 100 distinct providers. Features: virtual keys, budget caps, fallbacks, load balancing, semantic caching, logging integrations. 54k+ GitHub stars as of July 2026. Self-host via Docker; enterprise SaaS tier available. Security note: a critical pre-auth SQL injection in the proxy (CVE-2026-42208, CVSS 9.3) was disclosed in April 2026 and exploited within days — fixed in v1.83.7+; pin to a patched release. Source: github.com/BerriAI/litellm.

**OpenRouter** — hosted SaaS aggregator (no self-host option). One OpenAI-compatible API reaching 400+ models from 70+ providers. Per-token rates match provider-published prices (no per-token markup), but a platform fee (~5.5%, minimum $0.80) applies on credit purchases. Includes an **Auto Router** (powered by NotDiamond) that selects the best model per prompt automatically. Useful for latency-based and cost-based routing without running infrastructure. Source: openrouter.ai/docs.

**Portkey** — open-source AI gateway (Apache 2.0, gateway 2.0 fully open-sourced March 2026; github.com/Portkey-AI/gateway) with a managed SaaS cloud option. Routes to 250+ providers; includes semantic caching, guardrails, RBAC, observability, and MCP gateway support. Acquired by Palo Alto Networks (completed May 29, 2026); Portkey's gateway now serves as the AI Gateway layer for Palo Alto's Prisma AIRS platform. The open-source routing engine remains on GitHub under Apache 2.0; managed-tier features (observability storage, semantic caching at scale, team RBAC) stay cloud-only. Source: portkey.ai/docs.

**Cloudflare AI Gateway** — hosted SaaS (no self-host; Cloudflare-infrastructure only). Free tier available. One URL change routes any supported provider through Cloudflare's edge. Features: caching, rate limiting, logging, guardrails, request retry/fallback. Added a unified REST API (api.cloudflare.com) in May 2026. Source: developers.cloudflare.com/ai-gateway.

**Helicone** — open-source LLM observability platform and AI gateway (github.com/Helicone/helicone; YC W23). Integration model: change one URL. Provides request logging, cost tracking, caching, rate limiting, and agent trace inspection. Self-host via Docker/Helm or use the SaaS cloud (10k requests/month free). Acquired by Mintlify (March 3, 2026); now in maintenance mode — security patches, bug fixes, and new-model support only, no active feature development.

**Kong AI Gateway** — AI routing and governance layer built on top of Kong Gateway (the established API gateway). Adds AI-specific plugins: AI Proxy, AI Proxy Advanced (multi-provider load balancing), PII redaction, allow/deny lists for prompts. Kong AI Gateway 3.14 (April 2026) added "Kong Agent Gateway," making Agent-to-Agent (A2A) governance generally available alongside the LLM and MCP layers. Open-source core (Apache 2.0); enterprise tier available. Source: developer.konghq.com/ai-gateway.

**RouteLLM** — open-source model router framework (Apache 2.0; github.com/lm-sys/routellm; by LMSYS / Anyscale). Trains classifiers on human preference data to predict which model will produce better output. Routes simpler queries to cheap models, complex ones to capable models. Exposes an OpenAI-compatible server. Not a full gateway (no caching, key management, etc.) — a focused cost-based routing layer.

## Tradeoffs

- **Added latency** — the gateway is an extra network hop; well-operated gateways add <10 ms but under load this can grow.
- **New failure point** — if the gateway is down, all model calls fail. Self-hosted gateways require you to operate and scale them; SaaS gateways transfer that risk to the vendor.
- **Data privacy** — all prompts and responses pass through the gateway operator's infrastructure. For sensitive workloads, prefer self-hosted gateways or providers with strong data-processing agreements.
- **Vendor lock-in vs self-hosting** — SaaS gateways are operationally lightweight but create a dependency; self-hosted gateways (LiteLLM, Portkey, Kong) give full control at the cost of infrastructure work.

Cross-links: gateway logs are a primary telemetry source for agent observability (/resources/agent-observability); semantic caching at the gateway layer is a form of context reuse (/resources/agent-memory-context); routing to self-hosted models is a key gateway use case (/resources/open-weight-models-for-agents).

## Verified sources

- LiteLLM GitHub (BerriAI): https://github.com/BerriAI/litellm
- LiteLLM proxy docs: https://docs.litellm.ai/docs/simple_proxy
- OpenRouter docs: https://openrouter.ai/docs
- OpenRouter Auto Router docs: https://openrouter.ai/docs/guides/routing/routers/auto-router
- Portkey gateway GitHub (Portkey-AI): https://github.com/Portkey-AI/gateway
- Portkey docs: https://portkey.ai/docs
- Cloudflare AI Gateway overview: https://developers.cloudflare.com/ai-gateway/
- Cloudflare AI Gateway REST API changelog (May 2026): https://developers.cloudflare.com/changelog/post/2026-05-21-rest-api/
- LiteLLM CVE-2026-42208 advisory: https://docs.litellm.ai/blog/cve-2026-42208-litellm-proxy-sql-injection
- Palo Alto Networks completes Portkey acquisition (May 2026): https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-completes-acquisition-of-portkey-to-secure-ai-agents
- Helicone GitHub: https://github.com/Helicone/helicone
- Helicone joins Mintlify (March 2026): https://www.helicone.ai/blog/joining-mintlify
- Kong AI Gateway docs: https://developer.konghq.com/ai-gateway/
- RouteLLM GitHub (lm-sys): https://github.com/lm-sys/routellm


Code Execution Sandboxing for Agents

Isolation spectrum from language sandboxes to microVMs, WebAssembly as a portable sandbox, and a verified comparison of hosted agent-sandbox APIs — for agents that need to run model-generated code safely.

Running model-generated code is arbitrary code execution. Without isolation, a single malicious or buggy output can read host secrets, exfiltrate data, pivot to other tenants, or destroy infrastructure. Sandboxing is not optional for production agents that execute generated code or computer-use actions. See also: /resources/agentic-security-checklist (§7 — output and action sandboxing) and /resources/computer-use-browser-automation.

## Key facts

- Isolation for model-generated code runs on a spectrum from weakest to strongest: in-process language sandboxes, OS containers, hardened container runtimes, microVMs, and full VMs — each stronger layer costs more in startup time or overhead.
- Because a Docker container still shares the host kernel, a single kernel-level exploit can let malicious code climb out of it, so plain containers alone are not a strong enough boundary for code you don't trust.
- Hardened runtimes like gVisor intercept syscalls in user space and Kata Containers run each container in its own lightweight VM, eliminating or reducing the shared-kernel attack surface.
- MicroVMs such as Firecracker and Cloud Hypervisor boot a minimal hardware-virtualized kernel in well under 200 ms, delivering very strong isolation at latency closer to a container than a full VM.
- WebAssembly modules can only touch the filesystem or network if the embedding host opts them into that capability, though this deny-by-default model suits compiled languages much better than Python, whose WASM build also runs several times slower than native.
- Hosted agent-sandbox APIs (E2B, Modal, Daytona, Cloudflare Sandbox, Northflank, Vercel Sandbox) build on these same isolation layers so teams don't have to operate the infrastructure themselves.
- Regardless of isolation layer, harness-level hardening means blocking network egress by default, dropping privileges, wiping the filesystem between runs, capping CPU/memory/time, keeping host secrets out of the sandbox entirely, and validating whatever the sandbox hands back.

## The isolation spectrum

Weakest to strongest. Each layer adds real isolation at the cost of startup time, resource overhead, or surface complexity.

### 1. In-process language sandboxes

**What they are:** code is restricted at the language level before it runs — no separate process, no OS boundary.

- **RestrictedPython** (github.com/zopefoundation/RestrictedPython) — replaces Python's `compile()` to produce a restricted AST that disallows dangerous builtins and attribute access. The maintainers explicitly state it is not a full sandbox and must be combined with other controls; effectiveness depends on guard-function configuration.
- **Pyodide / WASM** — CPython compiled to WebAssembly and run inside the browser's or Node's WASM runtime. Network access and filesystem access are unavailable by default, so the WASM sandbox provides structural isolation without configuration. See §3 for WASM details.

**Isolation strength:** weak. Language sandboxes have no OS-level boundary. They are defeated by native extensions, JIT vulnerabilities, or overlooked builtins. Use only for very-low-risk inputs or as a first filter in a layered stack.

### 2. OS containers (Docker / standard runtimes)

**What they are:** Linux namespaces (PID, net, mount, UTS, IPC) plus cgroups isolate a process from the host filesystem and network. Docker is the dominant packaging and runtime implementation.

**Isolation strength:** moderate. Containers share the host kernel. A kernel exploit inside a container can escape to the host. For internal developer tooling or low-privilege workloads this is often acceptable, but standard Docker is NOT a strong security boundary against untrusted or adversarially generated code. The attack surface is the entire Linux kernel syscall table.

**Common misconception:** "We run it in Docker so it is safe." A container restricts what the workload *can see*, not what kernel vulnerabilities it can trigger. Isolate untrusted code at a higher level.

### 3. Hardened container runtimes

**What they are:** drop-in replacements for the container runtime that add a security layer between the container and the host kernel — without requiring full VM provisioning.

- **gVisor** (gvisor.dev) — developed by Google and open-sourced under Apache 2.0. Implements ~200 Linux syscalls in a user-space Go process called the Sentry. Application syscalls are intercepted by Sentry rather than reaching the host kernel directly; Sentry in turn makes a minimal set of host syscalls. To escape a gVisor sandbox, an attacker must simultaneously exploit Sentry and the host kernel, which share no code. Integrates with Docker and Kubernetes via the `runsc` runtime. Used by Google Cloud Run, OpenAI's Code Interpreter, and Modal.

- **Kata Containers** (katacontainers.io) — open-source project under the OpenInfra Foundation. Each container runs inside its own lightweight VM with a dedicated kernel. Kata supports multiple hypervisors as backends: Firecracker, Cloud Hypervisor, and QEMU/KVM. From the orchestration layer (Kubernetes, containerd) it looks like a standard container. Used by Northflank's sandbox platform.

**Isolation strength:** strong. Kernel attack surface is either eliminated (gVisor) or reduced to the hypervisor boundary (Kata). Startup overhead: gVisor adds milliseconds; Kata adds a VM boot (100–300 ms depending on hypervisor).

### 4. MicroVMs

**What they are:** purpose-built VMMs (Virtual Machine Monitors) that boot a minimal Linux kernel in a hardware-virtualized VM in under 200 ms, with a minimal device model and small memory footprint.

- **Firecracker** (github.com/firecracker-microvm/firecracker) — open-sourced by Amazon Web Services under Apache 2.0. The isolation technology behind AWS Lambda and AWS Fargate. Boots microVMs in ~125 ms, uses <5 MiB overhead per VM, and supports creating up to 150 microVMs/second on one host. Uses a jailer process and seccomp filters in production. Also used by E2B, Vercel Sandbox, and as a Kata Containers backend.

- **Cloud Hypervisor** (github.com/cloud-hypervisor/cloud-hypervisor) — open-source VMM written in Rust, hosted under the Linux Foundation. Targets modern cloud workloads with minimal device emulation, low latency, and small memory footprint. Memory and thread safety via Rust reduces VMM attack surface. Supported by the Kata Containers project as a hypervisor backend; used by Northflank.

**Isolation strength:** very strong. Hardware virtualization boundary separates each workload's kernel from the host. Industry-standard for multi-tenant serverless infrastructure.

### 5. Full VMs

Conventional VMs (KVM/QEMU, Hyper-V, VMware) provide the strongest isolation at the cost of the highest startup time (seconds to minutes) and resource overhead. Rarely the right choice for agent code execution where fast ephemeral sandboxes are needed; microVMs deliver equivalent security with orders-of-magnitude better latency.

## WebAssembly as a portable sandbox

WebAssembly (WASM) modules run in a capability-based, deny-by-default sandbox. A module cannot access memory outside its own linear memory, cannot make syscalls directly, and cannot use network or filesystem unless the host explicitly grants those capabilities.

- **Wasmtime** (wasmtime.dev) — production-grade WASM runtime by the Bytecode Alliance, written in Rust. Implements WASI (WebAssembly System Interface), which defines capability-based access to filesystem, network, clocks, and random — each must be explicitly granted by the host. A Wasmtime module with no imports has zero host access. Wasmtime is the dominant server-side and edge WASM runtime.

- **Pyodide** (github.com/pyodide/pyodide) — CPython compiled to WASM; runs in the browser or Node.js WASM sandbox. Supports NumPy, Pandas, and other C-extension packages via precompiled wheels. Network and filesystem are unavailable by default. Runs ~3–5× slower than native CPython. Ideal for in-browser Python REPL and lightweight sandboxed computation without server infrastructure.

WASM's limitation: compiled languages (Rust, C, Go) map well to WASM; Python via Pyodide is usable for many agent tasks but performance-sensitive workloads or native C extensions without precompiled wheels may not run. For general agent code execution, microVM-backed APIs (below) are typically the right default.

## Hosted agent-sandbox APIs

For agents that need to execute code without managing isolation infrastructure, these APIs provide sandboxed environments callable from agent code. All are web-verified as of June 2026.

| Product | Isolation | Cold start | GPU | Key differentiator |
|---|---|---|---|---|
| **E2B** (e2b.dev) | Firecracker microVM | ~150 ms | No | Agent-first SDK; Python/JS; MCP integration; free tier; production references (Perplexity, Manus) |
| **Modal** (modal.com) | gVisor containers | Fast | Yes (A100, H100) | 50k–100k concurrent sandboxes; GPU access; Lovable and Quora in production |
| **Daytona** (daytona.io) | Sysbox containers (user-namespace isolation) | <90 ms | No | Open-source; stateful persistent workspace; sub-90 ms via pre-warmed pools; $24M Series A Feb 2026 |
| **Cloudflare Sandbox** (developers.cloudflare.com/sandbox) | Containers (GA Apr 2026) + Dynamic Workers V8 isolates (beta) | ms (isolates) / container | No | Two-tier: full Linux containers via Sandbox SDK + isolate-based Dynamic Workers (100× faster); edge-distributed |
| **Northflank** (northflank.com) | Kata Containers (Cloud Hypervisor) + gVisor | — | No | Only platform offering both Kata and gVisor; BYOC (AWS/GCP/Azure); unlimited sessions; 2M+ isolated workloads/month |
| **Vercel Sandbox** (vercel.com/sandbox) | Firecracker microVM | — | No | GA (Jan 2026); free; 45 min–5 hr session cap; backed by Vercel Fluid compute; open-source Open Agents stack |

Cloudflare Sandbox SDK note: the sandbox itself is not deprecated, but several legacy SDK features are — HTTP and WebSocket transports are removed from SDK versions released after July 9, 2026 (migrate to the RPC transport), `exposePort()` is superseded by Cloudflare Tunnel, the desktop demo feature was removed in 0.10.2, and buffered read/write/exec APIs are consolidating onto streaming defaults. See the 2026 deprecation migration guide in the sources below.

### Provider built-ins

- **OpenAI Code Interpreter** (Responses API, developers.openai.com) — runs Python in gVisor-backed sandboxed containers. Available as the `code_interpreter` tool in the Responses API (added May 2025; Assistants API deprecated, shutdown August 26, 2026). Containers expire after 20 minutes of inactivity; supports file upload and download.

- **Anthropic code execution tool** (docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool) — runs Python and Bash in Anthropic's sandboxed container. Per Anthropic's migration guidance, the current server-tool version is `code_execution_20260521` (earlier `code_execution_*` versions should be migrated to it); `code_execution_20260120` is the variant that adds REPL state persistence and programmatic tool calling from within the sandbox, supported on Claude Opus 4.5+, Sonnet 4.5+.

## Hardening checklist

Regardless of the isolation layer, apply these controls at the harness level:

- **No network egress by default.** Block all outbound connections; allowlist specific hosts only (package registries, APIs the agent legitimately needs). Unrestricted egress allows data exfiltration and SSRF attacks against internal services.
- **Drop privileges.** Run sandbox processes as non-root; apply seccomp filter to restrict syscall surface (Firecracker's jailer does this automatically; configure explicitly for gVisor/container-based options).
- **Ephemeral filesystem.** Give each sandbox execution a fresh, ephemeral filesystem. Do not persist state between unrelated runs unless explicitly designed for it.
- **Resource limits.** Cap CPU time, memory, and wall-clock timeout per execution. Unbounded resource usage enables denial-of-service against the host infrastructure.
- **No host secrets in the sandbox.** Never inject API keys, database credentials, or other secrets into sandbox environment variables. Resolve secrets in the harness layer and pass only the minimum necessary result. See /resources/agentic-security-checklist §5.
- **Treat sandbox output as untrusted.** Validate and sanitize all output from a sandbox before using it in subsequent agent steps. A malicious payload could attempt prompt injection through generated output. See /resources/agentic-security-checklist §1.

## Verified sources

- Firecracker GitHub (AWS, Apache 2.0): https://github.com/firecracker-microvm/firecracker
- Firecracker — AWS Lambda and Fargate origin: https://aws.amazon.com/blogs/opensource/firecracker-open-source-secure-fast-microvm-serverless/
- gVisor security model: https://gvisor.dev/docs/architecture_guide/security/
- gVisor — What is gVisor: https://gvisor.dev/docs/
- Kata Containers GitHub: https://github.com/kata-containers/kata-containers
- Cloud Hypervisor GitHub: https://github.com/cloud-hypervisor/cloud-hypervisor
- Wasmtime security docs: https://docs.wasmtime.dev/security.html
- Pyodide GitHub: https://github.com/pyodide/pyodide
- RestrictedPython GitHub: https://github.com/zopefoundation/RestrictedPython
- E2B docs: https://e2b.dev/docs
- Modal — code execution sandboxes for AI agents (2026): https://modal.com/resources/best-code-execution-sandboxes-ai-agents
- Daytona homepage: https://www.daytona.io/
- Daytona security model (Sysbox runtime): https://www.daytona.io/docs/en/security-exhibit/
- Cloudflare Sandbox SDK docs: https://developers.cloudflare.com/sandbox/
- Cloudflare Sandboxes GA (April 2026): https://developers.cloudflare.com/changelog/post/2026-04-13-containers-sandbox-ga/
- Cloudflare Sandbox SDK 2026 deprecations (announced June 9, 2026): https://developers.cloudflare.com/changelog/post/2026-06-09-deprecating-sandbox-sdk-features/
- Cloudflare Sandbox SDK 2026 deprecation migration guide: https://developers.cloudflare.com/sandbox/guides/2026-deprecation/
- Northflank — how to sandbox AI agents: https://northflank.com/blog/how-to-sandbox-ai-agents
- Vercel Sandbox docs: https://vercel.com/docs/sandbox
- Vercel Sandbox GA (Jan 2026): https://vercel.com/changelog/vercel-sandboxes-ga
- OpenAI Code Interpreter tool: https://developers.openai.com/api/docs/guides/tools-code-interpreter
- Anthropic code execution tool: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool


Guardrails and Safety Filters for Agents

Runtime input/output/action controls that enforce policy independently of the model — tooling landscape, techniques, and layering guidance.

Guardrails are runtime checks that sit around — not inside — the model and enforce policy regardless of what the model itself produces. They complement but do not replace model-level safety training. See also: /resources/agentic-security-checklist for the threat-surface checklist that motivates these controls, and /resources/agent-control-specification for a newer, cross-framework specification that formalizes this input/output/action checkpoint pattern into a portable, vendor-neutral contract.

## Key facts

- Guardrails wrap the model rather than change it, enforcing policy no matter what the model outputs, and work alongside — not as a substitute for — built-in model safety training.
- Controls apply at three checkpoints: before the request reaches the model, after it replies but before the caller sees it, and immediately before any tool call fires — each screening for a different risk (injections/PII going in, harmful content or format errors coming out, unapproved or irreversible actions before execution).
- Detection techniques trade speed for flexibility: pattern-matching validators are instant but miss meaning-based violations, fine-tuned classifiers cover known harm categories quickly, a second model acting as judge handles nuance at higher cost, and grammar-constrained decoding removes format failures by restricting which tokens can even be generated.
- Because no single check catches everything — prompt injection especially has no complete fix — the recommended approach stacks cheap filters underneath slower, more thorough ones.
- The landscape spans open-weight classifiers from Meta, Google, IBM, and Alibaba/Qwen; open-source frameworks such as NeMo Guardrails and Guardrails AI; and vendor APIs from OpenAI, Microsoft, and Lakera (now part of Check Point).
- Beyond picking tools, mature deployments layer input/output/action coverage together, default to blocking when a check is uncertain, log every guardrail verdict against the run's trace ID, continually tune for false-positive/negative rates, and still require human sign-off for irreversible actions.

## Three guardrail positions

**Input guardrails** run before the model sees a request. Targets: jailbreak and prompt-injection attempts, PII in user turns, off-topic or policy-violating content. A classifier or rule fires here and can block, rewrite, or flag the request.

**Output guardrails** run after the model responds, before the response reaches the caller. Targets: harmful or policy-violating content, PII in generated text (redact before returning), schema or format violations, and groundedness/hallucination checks (does the answer stay within the supplied context?).

**Action guardrails** run before a tool call executes. Targets: tool calls outside an allowlist, calls that exceed defined parameter ranges, and irreversible or high-stakes operations that require human approval. See /resources/agentic-security-checklist §7 for the human-in-the-loop gate pattern.

## Techniques

- **Classifier-based filters** — a fine-tuned model labels input or output as safe/unsafe per a taxonomy. Fast and accurate for known harm categories; needs retraining as taxonomy evolves.
- **Regex / deterministic validators** — pattern matching for PII (credit-card numbers, email addresses), format enforcement (JSON schema, date formats). Zero latency; brittle for semantic violations.
- **LLM-as-judge guardrails** — a second LLM evaluates the primary model's output for policy compliance or groundedness. High flexibility; higher latency and cost.
- **Constrained decoding** — grammar-constrained generation forces the model to emit only tokens that satisfy a format specification at decode time. Eliminates format guardrail failures at the source. See /resources/reliable-tool-calling for structured-output details.

Defense-in-depth principle: stack multiple techniques. A regex catches known PII patterns cheaply; a classifier catches semantic violations the regex misses; an LLM-as-judge catches subtler policy issues. No single guardrail is sufficient, and prompt injection in particular has no complete solution — attackers can craft payloads that evade any single detector.

## Tooling landscape

**Llama Guard 4** (Meta, open-weight) — 12B multimodal safety classifier pruned from Llama 4 Scout. Classifies both prompt and response against the MLCommons hazards taxonomy; supports text and multiple images. Released April 2025. Model card: huggingface.co/meta-llama/Llama-Guard-4-12B.

**Llama Prompt Guard 2** (Meta, open-weight) — lightweight BERT-style (DeBERTa) classifiers (22M and 86M params) for detecting direct jailbreaks and prompt-injection attacks. Outputs benign/malicious label; 512-token context. Model card: huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M.

**NeMo Guardrails** (NVIDIA, open-source, Apache 2.0) — Python toolkit for adding programmable guardrails to LLM conversational systems. Intercepts input and output; policies expressed in Colang configuration; integrates with multiple embedding providers. GitHub: github.com/NVIDIA-NeMo/Guardrails.

**Guardrails AI** (guardrails-ai, open-source) — Python/JS framework for specifying and enforcing structure, type, and semantic constraints on LLM outputs. Includes a Guardrails Hub of pre-built validators; supports re-asking on failure. GitHub: github.com/guardrails-ai/guardrails.

**OpenAI Moderation API + Agents SDK guardrails** (OpenAI, vendor/SaaS) — the Moderation API is a free endpoint that classifies text (and images) for harmful content across categories (hate, harassment, self-harm, etc.). Moderation scores can also be requested inline with Responses API calls. The OpenAI Agents SDK exposes explicit input and output guardrail hooks that run per tool invocation. Docs: platform.openai.com/docs/guides/moderation.

**Azure AI Content Safety / Prompt Shields** (Microsoft, vendor/SaaS) — Content Safety API covers text and image harm categories. Prompt Shields (GA) detects user-prompt injection attacks and document attacks (indirect prompt injection embedded in retrieved content), with a Spotlighting capability announced at Build 2025. Docs: learn.microsoft.com/en-us/azure/ai-services/content-safety/.

**ShieldGemma** (Google, open-weight) — safety classifiers built on Gemma 2 (text, 2B/9B/27B params) covering four harm categories. ShieldGemma 2 (April 2025) is a 4B model built on Gemma 3 that adds image safety classification. Model card: huggingface.co/google/shieldgemma-9b.

**Granite Guardian** (IBM, open-weight) — safety models fine-tuned from IBM Granite. Latest: Granite Guardian 4.1 8B (April 2026), which adds improved bring-your-own-criteria (BYOC) support for custom judging criteria beyond pre-baked safety and hallucination detectors. Model card: huggingface.co/ibm-granite/granite-guardian-4.1-8b.

**Qwen3Guard** (Alibaba/Qwen, open-weight, Apache 2.0) — multilingual safety classifier series (0.6B / 4B / 8B) released September 2025, covering 119 languages. Two variants: Qwen3Guard-Gen for full-context generative classification and Qwen3Guard-Stream for token-level real-time streaming moderation. Model cards: huggingface.co/collections/Qwen/qwen3guard.

**Lakera Guard** (Check Point/Lakera, vendor/SaaS) — real-time API for detecting prompt injection, jailbreaks, and data leakage; claims sub-50 ms latency. Lakera was acquired by Check Point Software Technologies (announced September 2025, completed November 2025); now part of Check Point's AI security platform. Docs: docs.lakera.ai/guard.

## Practical guidance

- **Layer multiple guardrails.** Input + output + action coverage at minimum. A regex pre-filter reduces classifier load; a classifier catches what regex misses.
- **Fail closed on high-stakes actions.** If a guardrail errors or is inconclusive, block or escalate — do not default to allowing the action.
- **Log guardrail decisions.** Record every guardrail verdict (allowed/blocked, score, rule fired) alongside the trace ID for the agent run. See /resources/agent-observability for the broader observability pattern.
- **Measure false-positive and false-negative rates.** Guardrails that block too much degrade usability; guardrails that miss too much provide false confidence. Tune thresholds against a representative sample of real traffic.
- **Keep humans in the loop for irreversible actions.** No guardrail stack eliminates risk entirely, especially for prompt injection. For financial transfers, external communications, and data deletion, require explicit human confirmation regardless of guardrail output.

## Verified sources

- Llama Guard 4 model card (Meta/HuggingFace): https://huggingface.co/meta-llama/Llama-Guard-4-12B
- Llama Prompt Guard 2 model card (Meta/HuggingFace): https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M
- NeMo Guardrails GitHub (NVIDIA): https://github.com/NVIDIA-NeMo/Guardrails
- Guardrails AI GitHub: https://github.com/guardrails-ai/guardrails
- OpenAI Moderation API docs: https://platform.openai.com/docs/guides/moderation
- OpenAI Agents SDK guardrails: https://openai.github.io/openai-agents-python/guardrails/
- Azure AI Content Safety / Prompt Shields (Microsoft Learn): https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection
- ShieldGemma model card (Google/HuggingFace): https://huggingface.co/google/shieldgemma-9b
- ShieldGemma 2 on HuggingFace: https://huggingface.co/google/shieldgemma-2-4b-it
- Granite Guardian 4.1 8B model card (IBM/HuggingFace): https://huggingface.co/ibm-granite/granite-guardian-4.1-8b
- Lakera Guard docs (Check Point/Lakera): https://docs.lakera.ai/guard
- Check Point acquires Lakera (press release): https://www.checkpoint.com/press-releases/check-point-acquires-lakera-to-deliver-end-to-end-ai-security-for-enterprises/
- Qwen3Guard GitHub (Alibaba/Qwen): https://github.com/QwenLM/Qwen3Guard
- Qwen3Guard technical report (arXiv): https://arxiv.org/abs/2510.14276


Embeddings and Vector Search for Agents

How to pick an embedding model, understand distance metrics, choose an ANN index type, and operate a vector store reliably in agent retrieval pipelines.

Embeddings map text (or other content) to dense numeric vectors so that semantic similarity can be measured by vector distance. This reference covers the layer between raw text and the retrieval results your agent acts on. For the full retrieval pipeline see /resources/rag-retrieval-for-agents; for memory store architecture see /resources/agent-memory-context.

## Embedding model dimensions and selection criteria

Key parameters to evaluate before choosing a model:

- **Dimensionality** — typical range 256–3072. Higher dimensions can capture more nuance but cost more storage and compute. Matryoshka Representation Learning (MRL, Kusupati et al., NeurIPS 2022, arXiv:2205.13147) trains embeddings so that truncating to fewer dimensions still yields useful representations — use MRL-trained models to right-size vectors at query time without re-embedding.
- **Max input tokens** — models differ widely (256–128k tokens). A model that silently truncates long documents will produce misleading embeddings.
- **Language coverage** — English-only models underperform on multilingual corpora. Verify coverage for your target languages.
- **Domain fit** — general-purpose MTEB scores may not reflect performance on code, legal, medical, or financial text. Benchmark on your own data.
- **Cost / quality / latency tradeoff** — API-hosted models charge per token; open-weight models traded compute for zero marginal cost. Quantized open-weight models reduce memory at the cost of some recall.

## Standard benchmark: MTEB / MMTEB

The **Massive Text Embedding Benchmark (MTEB)** — maintained by the embeddings-benchmark organization on HuggingFace (huggingface.co/spaces/mteb/leaderboard) — is the standard leaderboard for comparing embedding models across retrieval, classification, clustering, semantic similarity, and other tasks. It covers 112+ languages and 5,000+ submissions.

MMTEB (Massive Multilingual Text Embedding Benchmark, arXiv:2502.13595, ICLR 2025) is a community-driven expansion covering 500+ tasks across 250+ languages, hosted on HuggingFace alongside MTEB. Use MMTEB scores when multilingual recall matters.

Do not treat leaderboard rankings as permanent. Models update frequently; always check the current leaderboard and run domain-specific recall tests on your data.

## Notable embedding model families (as of mid-2026)

Listed as available/notable — rankings shift; verify current MTEB position before choosing:

**API / hosted:**
- **OpenAI text-embedding-3-small / text-embedding-3-large** — 1536 and 3072 default dimensions; both support MRL-style dimension reduction via the `dimensions` parameter (openai.com).
- **Cohere Embed v4** — multimodal (text + images), 128k token context, supports MRL dimensions (256/512/1024/1536), 100+ languages (cohere.com).
- **Voyage AI (voyage-3.5 / voyage-3.5-lite)** — strong retrieval scores; MRL dimensions (256/512/1024/2048); multilingual support (voyageai.com).
- **Google Gemini Embedding (gemini-embedding-001)** — up to 3072 dimensions, MRL-truncatable, 100+ languages; text-embedding-004 deprecated as of Jan 2026 (ai.google.dev).

**Open-weight:**
- **BGE / BAAI (e.g., BGE-M3)** — multi-functionality (dense, sparse, multi-vector), multilingual, widely deployed, top open-weight download counts on HuggingFace (huggingface.co/BAAI).
- **E5 / multilingual-E5** (Microsoft) — strong multilingual retrieval; multiple sizes available (huggingface.co/intfloat).
- **Nomic Embed** — long-context, Apache 2.0, reproducibly trained (nomic.ai; arxiv:2402.01613).
- **Jina Embeddings** — extended context; self-hostable (jina.ai).
- **Qwen3 Embedding** (Alibaba/QwenLM) — 0.6B/4B/8B sizes; Apache 2.0; ranked highly on MTEB multilingual as of June 2025 (github.com/QwenLM).

Cross-reference open-weight model infrastructure at /resources/open-weight-models-for-agents.

## Distance metrics

| Metric | Formula basis | Best for |
|--------|--------------|---------|
| **Cosine similarity** | Angle between vectors | Default for most retrieval; normalizes magnitude |
| **Dot product** | Magnitude × angle | Equivalent to cosine when vectors are unit-normalized; faster |
| **Euclidean (L2)** | Absolute distance | Useful when magnitude carries information (e.g., sparse embeddings) |

**Normalization caveat:** most retrieval models produce unit-normalized vectors by default, making cosine and dot product equivalent. If you normalize, dot product is cheaper to compute. Verify your model's output normalization before choosing.

## ANN index types

Approximate Nearest Neighbor (ANN) indexes trade exact recall for speed. The main types:

- **Flat (brute-force)** — exact exhaustive scan; 100% recall; scales linearly with corpus size. Use only for small corpora (<100k vectors) or as a recall baseline.
- **IVF (Inverted File Index)** — k-means clusters vectors at build time; at query time only nearby clusters are searched (multi-probe tunable). Faster build than HNSW; lower memory; requires a training step. Implemented in FAISS (github.com/facebookresearch/faiss) and pgvector (IVFFlat).
- **HNSW (Hierarchical Navigable Small World graphs)** — multi-layer proximity graph; logarithmic search complexity; superior speed-recall tradeoff at query time; slower to build and more memory than IVF; no training step required. See Malkov & Yashunin, arXiv:1603.09320. Implemented in FAISS and pgvector.
- **Product Quantization (PQ) / Scalar Quantization (SQ)** — compression schemes that reduce vector storage by encoding sub-vectors with a codebook. Typically combined with IVF (IVFPQ) or HNSW. Reduces memory 4–32× at the cost of some recall. Use when corpus exceeds available RAM.

## Practical guidance

- **Match query and document models.** Embeddings from different models occupy different vector spaces — they are not comparable. Always embed queries with the same model used to index documents.
- **Re-embed on model change.** Embeddings are not portable across model versions. Upgrading an embedding model requires re-indexing the full corpus.
- **Normalize before cosine search.** If your vector store does not auto-normalize, do it at index and query time to avoid misleading distance scores.
- **Use quantization at scale.** PQ or SQ cuts memory 4–32× with modest recall loss. Benchmark recall on your own data before and after.
- **Benchmark recall on your own data.** Leaderboard scores are averages. Your domain, query style, and document length distribution all affect real-world recall.
- **Start with HNSW for online search; use IVFPQ for large memory-constrained corpora.**

## Verified sources

- MTEB leaderboard (HuggingFace): https://huggingface.co/spaces/mteb/leaderboard
- MMTEB paper (arXiv:2502.13595): https://arxiv.org/abs/2502.13595
- Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022, arXiv:2205.13147): https://arxiv.org/abs/2205.13147
- HNSW paper (Malkov & Yashunin, arXiv:1603.09320): https://arxiv.org/abs/1603.09320
- OpenAI embedding models (openai.com): https://openai.com/index/new-embedding-models-and-api-updates/
- Cohere Embed v4 announcement: https://cohere.com/blog/embed-4
- Voyage AI embedding docs: https://docs.voyageai.com/docs/embeddings
- Google Gemini Embedding GA (Google Developers Blog): https://developers.googleblog.com/gemini-embedding-available-gemini-api/
- Qwen3 Embedding (QwenLM GitHub): https://github.com/QwenLM/Qwen3-Embedding
- Nomic Embed paper (arXiv:2402.01613): https://arxiv.org/abs/2402.01613
- FAISS (Meta AI): https://github.com/facebookresearch/faiss
- pgvector (PostgreSQL vector extension): https://github.com/pgvector/pgvector


Agent Cost and Latency Optimization

Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).

Production agents are expensive and slow for a structural reason: agent loops *multiply* both cost and latency. A single model call is cheap; a 20-step agentic trajectory at 10k tokens per call is not. Optimization levers exist at four levels — token, request, model, and architecture — and the highest-ROI wins are almost always architectural.

## Key facts

- Because every step in a trajectory repeats the same spend, total cost and latency track how many steps a run takes far more than the price of any single call, and the available fixes cluster into four tiers running from individual tokens up to the whole architecture, with the top tier usually paying off the most.
- Fan-out drives spend fastest: dispatching several sub-agents that each make many calls multiplies total cost far beyond a single-agent, single-call baseline, so counting steps per task matters more than trimming individual calls.
- Caching a stable prompt prefix is typically the highest-leverage token-side fix when agents repeat the same system instructions or reference material call after call, and every major provider now applies some form of discount on cache hits, though the percentage and minimum prompt size differ per vendor.
- At the request level, an asynchronous batch endpoint substantially discounts workloads that can tolerate a same-day turnaround, streaming shortens perceived wait time without lowering spend, and running independent tool/sub-agent calls concurrently turns latency into a max instead of a sum.
- Model-level savings come from cascading cheap models before escalating to bigger ones, delegating deterministic sub-tasks to smaller or open-weight models, and dialing down reasoning-effort or thinking-budget parameters per task.
- The biggest architectural win is simply cutting the number of turns a trajectory takes, followed by caching whole responses for near-duplicate queries, stopping early once a good-enough answer exists, and pushing deterministic work like math or formatting out to code instead of the model.

## The compounding model

For a trajectory of N model calls:

- **Cost ≈ tokens_per_call × price_per_token × N** (fan-out multiplies spend linearly).
- **Latency ≈ per_call_latency × N + tool_io_latency** (steps are serial by default; parallelism is the main structural escape).

Fan-out is the dominant cost driver. A pipeline that spawns 5 sub-agents each making 10 calls multiplies spend by 50× relative to a single-agent 1-call solution. Measure steps-per-task before optimizing individual calls — see /resources/agent-observability (you cannot optimize what you do not measure; track cost and latency per span).

## Token-level levers

**Prompt caching (provider prefix caching)** is the highest-ROI single lever for agents with a stable system prompt or large reference context that repeats across calls:

- **Anthropic**: explicit cache control via `cache_control` breakpoints. Cache reads cost ~10% of standard input tokens (90% discount). Cache writes cost 1.25× standard input. TTL 5 minutes (extendable). Supported across the Claude 3/4 families.
- **OpenAI**: automatic for prompts >1 024 tokens on supported models (GPT-4o, o-series, GPT-5 family). Cache hits receive a 50% discount on input tokens. No explicit API knob required — prefix stability determines hit rate.
- **Google Gemini**: "implicit caching" enabled by default on Gemini 2.5 models. Minimum 1 024 tokens (Flash) or 2 048 tokens (Pro). Cache hits on Gemini 2.5 receive a 75% discount on cached tokens. Explicit context caching (named cache objects) is also available with configurable TTL.

To exploit caching: put stable content (system prompt, reference documents, few-shot examples) at the top of the context; vary only the dynamic suffix. Cross-link: /resources/agent-memory-context covers prompt caching as a context-management technique.

**Concise prompts and context pruning**: every unnecessary token in the context costs money on every call. Remove boilerplate, truncate stale history, and summarize instead of concatenating. Sliding-window and summarization strategies are covered in /resources/agent-memory-context.

**Structured outputs to avoid re-asks**: a malformed tool call or unparseable JSON response forces a retry, doubling token spend for that turn. Enable provider-level constrained decoding (OpenAI strict mode, Anthropic tool_choice forcing, Gemini ANY mode) to eliminate format retries. See /resources/reliable-tool-calling.

**Limit injected context**: retrieve only the chunks an agent actually needs (RAG top-k), not the entire knowledge base. See /resources/rag-retrieval-for-agents.

## Request-level levers

**Batch API (async, 50% discount)**: both Anthropic and OpenAI offer an asynchronous batch processing endpoint that processes requests within a 24-hour window in exchange for a 50% discount on all token costs (input and output). Verified for both providers as of June 2026. Use for non-latency-sensitive workloads: bulk document processing, nightly eval runs, data enrichment pipelines, and offline agentic sweeps. Batch and prompt caching discounts stack independently.

**Streaming for perceived latency**: for user-facing agents, enable streaming responses. The model starts returning tokens immediately; time-to-first-token drops significantly even when total generation time is unchanged. This does not reduce cost but reduces perceived latency, which matters for interactive agents.

**Parallelizing independent tool calls and sub-agents**: when an agent step fans out to multiple independent tool calls, run them in parallel (concurrent API calls or sub-agent threads) rather than sequentially. This reduces latency from sum-of-steps to max-of-steps. Most frameworks (LangGraph, OpenAI Agents SDK, CrewAI) support parallel tool execution natively. See /resources/multi-agent-orchestration-patterns for fan-out cost analysis.

## Model-level levers

**Model routing / cascades (cheap model first, escalate)**: route simple sub-tasks to a smaller, cheaper model and escalate to a larger model only when the cheaper model signals low confidence or the task matches known complexity thresholds. A well-tuned cascade can cut average cost per task substantially with minimal quality loss. See /resources/ai-gateways-llm-routing for routing infrastructure.

**Smaller / distilled models for sub-tasks**: use a large model for complex reasoning and orchestration, but delegate deterministic sub-tasks (classification, extraction, translation) to smaller distilled models. Open-weight models (Llama 4 Scout, Qwen 3.6, Phi-4-mini, Granite 4.1 3B) eliminate per-token vendor fees entirely for sub-tasks you can self-host. See /resources/open-weight-models-for-agents.

**Reasoning-effort controls**: providers expose knobs to trade reasoning depth for cost and latency:

- **OpenAI**: `reasoning_effort` parameter (values: `low`, `medium`, `high`, `xhigh` on newer models; `none`/`minimal` on some). Lower effort uses fewer reasoning tokens — faster and cheaper. Default is `medium` on GPT-5.5. Verified via OpenAI reasoning docs.
- **Anthropic**: extended thinking uses `budget_tokens` to set the maximum reasoning token budget. Newer Claude 4.x models support `type: "adaptive"` (the older `type: "enabled"` with explicit `budget_tokens` is deprecated but still functional). Reducing the budget cuts thinking cost directly.

Tune reasoning effort per task category — high effort for complex multi-step decisions, low/minimal for classification or extraction.

**Speculative decoding (self-hosted)**: an inference-level optimization where a small "draft" model generates candidate tokens that the large "verifier" model checks in parallel, rejecting tokens that deviate from its distribution. Reduces latency by 2–4× on CPU-bounded inference hardware with no quality loss. Relevant for self-hosted deployments; not controllable via hosted provider APIs. See /resources/open-weight-models-for-agents.

## Architecture-level levers

**Reduce steps (fewer agent turns)**: the single highest-leverage intervention. Every additional turn multiplies both cost and latency. Audit your trajectories for redundant turns: unnecessary clarification rounds, sequential tool calls that could be one batched call, and loops that could be replaced by a single tool with richer output. If an agent reliably completes a task in 4 turns, refactoring to 2 turns halves both cost and latency with no model change. See /resources/multi-agent-orchestration-patterns.

**Semantic caching of whole responses**: cache complete LLM responses for semantically equivalent queries. When a new query is within a configurable cosine-similarity threshold of a cached query, return the cached response without a model call. Tools: GPTCache (github.com/zilliztech/GPTCache), Redis with vector similarity, or LangChain's built-in semantic cache. Effective for agents with repetitive sub-tasks or high query overlap. Not effective for inherently unique or user-specific queries.

**Early-exit and termination guards**: add explicit completion detection so the agent stops as soon as it has a satisfactory answer rather than continuing for a fixed maximum step count. An agent that runs 20 steps when 6 would suffice wastes 14 turns of compute.

**Offload deterministic work to code and tools**: do not ask the model to do arithmetic, date calculations, regex matching, or JSON transformation — call a function instead. Model inference is the expensive operation; a Python function call costs microseconds. This also improves reliability: models make arithmetic errors; code does not.

## Measure first

No optimization is worth implementing without measurement. Instrument every agent run with span-level token counts, cost attribution, and latency per step before tuning. The highest-cost spans are rarely where intuition points. See /resources/agent-observability.

## Highest-ROI checklist (in approximate order)

- [ ] Instrument traces: cost and latency per span, steps per task.
- [ ] Count steps per trajectory — reduce unnecessary turns before anything else.
- [ ] Enable prompt caching for stable system-prompt prefixes.
- [ ] Parallelize independent tool calls and sub-agent delegations.
- [ ] Route simple sub-tasks to smaller/cheaper models.
- [ ] Use constrained decoding to eliminate format-retry turns.
- [ ] Shift non-latency-sensitive workloads to Batch API (50% discount).
- [ ] Tune reasoning_effort / budget_tokens per task category.
- [ ] Prune injected context to only what the current step needs.
- [ ] Add semantic caching for repeated sub-task queries.
- [ ] Offload deterministic computation to tool functions, not the model.

## Verified sources

- Anthropic prompt caching docs: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
- Anthropic Message Batches API docs: https://docs.anthropic.com/en/docs/build-with-claude/message-batches
- OpenAI prompt caching announcement: https://openai.com/index/api-prompt-caching/
- OpenAI Batch API pricing: https://openai.com/api/pricing/
- OpenAI reasoning effort docs: https://developers.openai.com/api/docs/guides/reasoning
- Google Gemini implicit caching announcement: https://developers.googleblog.com/gemini-2-5-models-now-support-implicit-caching/
- Google Gemini context caching docs: https://ai.google.dev/gemini-api/docs/caching
- Anthropic extended thinking (adaptive mode / budget_tokens): https://platform.claude.com/docs/en/build-with-claude/extended-thinking
- GPTCache (semantic whole-response caching): https://github.com/zilliztech/GPTCache


Voice and Realtime Agents

Architectures, vendor APIs, and open frameworks for real-time speech-to-speech AI agents — cascaded pipeline vs. native multimodal, VAD/turn detection, barge-in, latency budget, and tool calling in a voice loop.

Real-time voice agents are one of the fastest-growing deployment patterns in 2026. Two architectures dominate. Understanding the tradeoffs between them is the prerequisite for every vendor and framework choice downstream.

## Key facts

- Real-time voice agents split into two dominant architectures: a cascaded STT → LLM → TTS pipeline, and a single native speech-to-speech model that ingests and emits raw audio directly.
- The cascaded pipeline gives full control to swap any component — including running each stage on an open-weight model — but pays a latency and cost penalty from three sequential model calls per turn.
- Native speech-to-speech models offer lower end-to-end latency and richer built-in prosody/interruption handling, at the cost of vendor lock-in on the underlying model and expensive audio tokens; open-weight native speech-to-speech options are still limited as of mid-2026.
- Turn detection and barge-in/interruption handling are explicit engineering concerns in a cascaded pipeline but are handled inside the model for native speech-to-speech systems.
- Vendor realtime APIs from OpenAI, Google, Amazon, and xAI all now support mid-conversation tool/function calling alongside streaming audio.
- Transport choice tends to follow the use case — WebRTC for browser/mobile clients, WebSocket for server-to-server integration — with some vendors adding a SIP path for telephony.
- Some vendors design for drop-in compatibility with each other: xAI's realtime API matches the OpenAI Realtime API's specification, so existing OpenAI-built clients can point at it with minimal changes.

## Architecture 1: Cascaded pipeline (STT → LLM → TTS)

The classic pipeline chains three separate models:

1. **STT** — streaming speech-to-text converts the user's audio to a text transcript.
2. **LLM** — the transcript is fed to a language model, which produces a text reply (and may call tools).
3. **TTS** — the reply is synthesized back to audio.

A **VAD (Voice Activity Detection)** module sits upstream to detect when the user is speaking and trigger **end-of-turn detection** — the decision that the user has finished and the agent should respond. Between the models, **barge-in / interruption handling** flushes the TTS buffer and restarts the STT stage when the user speaks over the agent.

**Tradeoffs:**

| Dimension | Cascaded pipeline |
|---|---|
| Latency | Higher (three sequential models); target sub-second requires fast STT + cached LLM prefix + streaming TTS |
| Interruptibility | Requires explicit barge-in logic at each stage boundary |
| Emotion / prosody | TTS adds prosody; quality varies by provider |
| Cost | Pay for three separate model calls per turn |
| Control | High: swap any component independently; use any LLM |
| Open-weight path | Yes — each stage can run on open-weight models |

## Architecture 2: Native speech-to-speech (realtime multimodal models)

A single model ingests raw audio and outputs raw audio directly, without a text intermediate at the core inference step. Turn detection, interruption handling, and prosody are handled inside the model.

**Tradeoffs:**

| Dimension | Native speech-to-speech |
|---|---|
| Latency | Lower end-to-end (one model, streaming output) |
| Interruptibility | Built into the model; lower barge-in latency |
| Emotion / prosody | Richer; the model controls vocal tone end-to-end |
| Cost | Single model call, but audio tokens are expensive |
| Control | Lower: you cannot swap the underlying LLM independently |
| Open-weight path | Limited — open-weight native speech-to-speech models are still emerging as of mid-2026 |

## Vendor realtime APIs

All entries below are web-verified as of 2026-07-09.

### OpenAI Realtime API

A native speech-to-speech API. The original `gpt-realtime` (GA August 28, 2025) was superseded by **`gpt-realtime-2`** (May 7, 2026 — GPT-5-class reasoning, configurable reasoning effort, 128K context), then by **`gpt-realtime-2.1`** and mini variant **`gpt-realtime-2.1-mini`** (July 6, 2026 — improved alphanumeric recognition, noise/silence handling, and interruption behavior; ~25% lower p95 latency via caching). List pricing for 2.1 is unchanged from gpt-realtime-2 / the original mini (audio ≈$32/$64 per 1M input/output tokens full model, ≈$10/$20 mini) — no price cut shipped alongside 2.1, despite some chatter to that effect. The earlier `gpt-4o-realtime-preview` series is deprecated.

Transports: **WebRTC** (recommended for browsers and mobile — lower jitter, handles NAT traversal) and **WebSocket** (recommended for server-to-server). A **SIP** integration path is also available for telephony.

Supports streaming audio input and output, tool/function calling mid-conversation, VAD and server-side turn detection, and barge-in. Approximate glass-to-glass latency: 300–600 ms on subsequent turns.

Docs: platform.openai.com/docs/guides/realtime-webrtc, developers.openai.com/api/docs/guides/realtime-websocket, and developers.openai.com/api/docs/models/gpt-realtime-2.1

### Google Gemini Live API

A native speech-to-speech API with bidirectional streaming over **WebSocket**. The model processes audio input and returns audio output natively, without a text intermediate. GA model: Gemini 2.5 Flash (native audio), available via both Google AI for Developers and Vertex AI. A newer **Gemini 3.1 Flash Live** (preview, released March 26, 2026) adds sharper acoustic-nuance detection and lower latency, but as of this writing is available only via Google AI Studio — no Vertex AI availability or GA date has been announced.

Supports multimodal input (audio + video/screen), turn detection, barge-in, and function calling.

Docs: ai.google.dev/gemini-api/docs/live-api

### Amazon Nova Sonic

A native speech-to-speech model on Amazon Bedrock, announced April 2025. The current generation is **Amazon Nova 2 Sonic** (December 2025). Accessed via Bedrock's **bidirectional streaming API** (WebSocket). Also supports WebRTC via an AWS blog reference implementation.

Supports tool use, voice selection, interruption handling, and background-noise robustness. Integrates with Amazon Connect and telephony providers (Vonage, Twilio) and open frameworks including LiveKit and Pipecat.

Docs: docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html

### xAI Grok Voice Agent API

A realtime speech-to-speech API launched December 17, 2025. Uses **bidirectional WebSocket** streaming. Compatible with the OpenAI Realtime API specification, so clients built for OpenAI Realtime can point at xAI with minimal changes.

Features: custom VAD, Smart Turn end-of-turn detection, sub-1-second time-to-first-audio, 100+ language support with automatic detection. Also available via a native LiveKit plugin.

Docs: docs.x.ai/docs/guides/voice

## Open frameworks and orchestrators

### Pipecat (pipecat-ai)

Open-source Python framework (BSD-2-Clause) for building real-time voice and multimodal conversational agents, developed by Daily. Organizes processing as pipeline frames flowing through transport, STT, LLM, and TTS stages. Supports 20+ STT providers and 30+ TTS providers, plus direct integrations with native speech-to-speech services (OpenAI Realtime, Amazon Nova Sonic, Gemini Live).

Transports: WebRTC (Daily, LiveKit, SmallWebRTC), WebSocket, telephony. Handles VAD, turn detection, barge-in, and multi-agent coordination.

GitHub: github.com/pipecat-ai/pipecat

### LiveKit Agents

Open-source Python and TypeScript framework (Apache 2.0) for building realtime voice, video, and physical AI agents on top of the LiveKit WebRTC infrastructure. SDK v1.0 GA April 2025 (current line: v1.6.x as of mid-2026).

The 1.0 release replaced the older **VoicePipelineAgent** and **MultimodalAgent** classes (both now deprecated) with a single unified orchestrator, **AgentSession**, which covers cascaded (STT → LLM → TTS) and native speech-to-speech backends (e.g. OpenAI Realtime, Gemini Live) without changing application code when switching between them. Includes built-in turn detection, barge-in, native MCP tool support, and function calling. Bring-your-own STT, LLM, and TTS with no lock-in.

Docs: docs.livekit.io/agents

## STT and TTS component vendors

**STT:**

- **Deepgram** — streaming STT via WebSocket. Nova-3 is the flagship model (low WER). Flux is a conversational STT model with model-integrated end-of-turn detection and configurable turn-taking dynamics, designed specifically for voice agent pipelines. Docs: developers.deepgram.com

- **whisper.cpp** (ggml-org) — C/C++ port of OpenAI's Whisper ASR models; runs locally with no external dependencies. Supports GPU acceleration and VAD. Use for on-device or self-hosted STT when latency from network round-trips to a cloud STT API is a constraint. GitHub: github.com/ggml-org/whisper.cpp

**TTS:**

- **ElevenLabs** — streaming TTS via WebSocket. Eleven v3 model. Broadest language coverage (70+ languages), strong voice cloning. Streaming endpoint: `/v1/text-to-speech/{voice_id}/stream-input`. Docs: elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-input

- **Cartesia** — streaming TTS via WebSocket. **Sonic-3.5** (released June 2026) is the current flagship — approx. 40 ms time-to-first-audio on the Turbo variant, sub-100 ms on standard, native support for 42 languages. The prior Sonic 3 model remains available. Optimized for streaming latency at scale. Docs: docs.cartesia.ai

## Key concepts

- **VAD (Voice Activity Detection)** — a classifier that scores incoming audio for voice presence, typically using an energy threshold plus a voice classifier plus a minimum-duration guard. Determines when the user is speaking.
- **End-of-turn detection / endpointing** — deciding that the user has finished their utterance and the agent should respond. Can be energy-based (silence duration), STT-integrated (model signals completion), or model-based (an ML classifier predicts semantic completion rather than just silence). Model-based detection has lower latency than waiting for silence timeouts.
- **Barge-in / interruption handling** — detecting when the user speaks while the agent is responding, flushing the TTS output buffer, and restarting the listen/respond cycle. Failure to handle barge-in correctly is the most common CSAT failure in voice agents — distinguish genuine interruptions from backchannels ("uh-huh", "right") to avoid cutting off mid-sentence unnecessarily.
- **Latency budget** — target sub-600 ms glass-to-glass for natural-feeling turn-taking. In a cascaded pipeline, the budget is roughly: STT (50–150 ms) + LLM time-to-first-token (100–300 ms) + TTS time-to-first-audio (50–150 ms). Each component must be streaming — do not wait for full STT transcript before starting LLM inference. See /resources/agent-cost-latency-optimization.
- **Tool calling in a voice loop** — voice agents can call tools mid-conversation, but tool latency adds directly to voice latency. Keep tool calls under 200 ms; use speculative execution for predictable tool calls; return partial results to the model via streaming where possible. See /resources/reliable-tool-calling.

## Practical guidance

- **Measure end-to-end latency** per component (STT → LLM → TTS), not just total. Instrument each stage separately; bottlenecks are rarely where intuition points.
- **Handle interruptions before tuning latency** — a fast agent that cannot be interrupted is worse than a slightly slower one that can.
- **Keep tool calls fast** — tool latency is voice latency. Parallelize independent tool calls; cache results of stable lookups.
- **Plan for transcription errors** — cascade pipelines inherit STT errors. Design prompts and tool schemas to tolerate common transcription noise (homophones, dropped words).
- **Budget for always-on audio cost** — native speech-to-speech APIs price audio tokens at rates 3–10× higher than text tokens. Profile real traffic before committing to a pricing model.
- For latency optimization across the full agent stack, see /resources/agent-cost-latency-optimization.
- For tool-calling reliability in a voice loop, see /resources/reliable-tool-calling.
- For framework choices when adding voice to a multi-agent system, see /resources/agent-frameworks-compared.

## Verified sources

- OpenAI Realtime API (WebRTC): https://platform.openai.com/docs/guides/realtime-webrtc
- OpenAI Realtime API (WebSocket): https://developers.openai.com/api/docs/guides/realtime-websocket
- OpenAI gpt-realtime GA announcement: https://openai.com/index/introducing-gpt-realtime/
- OpenAI gpt-realtime-2.1 / gpt-realtime-2.1-mini announcement (July 6, 2026): https://community.openai.com/t/new-realtime-models-on-the-api-gpt-realtime-2-1-and-gpt-realtime-2-1-mini/1385896
- gpt-realtime-2.1 pricing unchanged from gpt-realtime-2/original mini (no cut): https://www.marktechpost.com/2026/07/06/openai-gpt-realtime-2-1-mini-reasoning-realtime-api/
- Google Gemini Live API overview: https://ai.google.dev/gemini-api/docs/live-api
- Google Gemini 3.1 Flash Live (preview, March 26, 2026): https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview
- Google Gemini Live API (WebSocket get started): https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket
- Amazon Nova Sonic announcement (April 2025): https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-sonic-speech-to-speech-conversations-bedrock/
- Amazon Nova 2 Sonic announcement (December 2025): https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-nova-2-sonic-real-time-conversational-ai
- Amazon Nova Sonic bidirectional streaming docs: https://docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html
- xAI Grok Voice Agent API announcement: https://x.ai/news/grok-voice-agent-api
- xAI Voice Agent API docs: https://docs.x.ai/docs/guides/voice
- Pipecat (pipecat-ai, BSD-2-Clause): https://github.com/pipecat-ai/pipecat
- LiveKit Agents docs: https://docs.livekit.io/agents/voice-agent/
- LiveKit Agents source (v1.6.x, Apache-2.0, confirms AgentSession as the current API): https://github.com/livekit/agents
- Deepgram Nova-3 and Flux STT docs: https://developers.deepgram.com/docs/models-languages-overview
- whisper.cpp (ggml-org, C/C++ Whisper port): https://github.com/ggml-org/whisper.cpp
- ElevenLabs TTS WebSocket API: https://elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-input
- Cartesia TTS WebSocket API: https://docs.cartesia.ai/api-reference/tts/websocket
- Cartesia Sonic-3.5 / Ink-2 launch (June 2026): https://cartesia.ai/launch


Web Data and Scraping for Agents

Tool landscape for agent web-data pipelines: reader/URL-to-Markdown APIs, crawl/scrape services, and search APIs — with MCP exposure, OSS/SaaS classification, and practical guidance.

Agents cannot use raw HTML efficiently: it bloats the context window, embeds navigation noise, and costs 3–10x more tokens than clean Markdown of the same content. A web-data layer transforms live web content into agent-consumable form. Three distinct jobs require different tools.

## Key facts

- Feeding an agent raw HTML instead of a converted, cleaned-up text format multiplies its token bill several times over for the same underlying content, which is the core reason a dedicated web-data transformation layer exists at all.
- Web-data tooling splits into three distinct jobs — reading a single known URL, crawling/scraping at scale, and searching the web — each best served by a different class of tool rather than one do-everything product.
- For single-URL reading, running your own open-source extractor costs nothing and keeps data in-house, but it has no way to get past a CAPTCHA, which is exactly what the hosted reader/scrape services are built to handle automatically.
- Some crawl/scrape services now expose payment or MCP integration directly — for example, tens of thousands of Apify Actors became payable per-run over x402 with no account needed, and several providers ship their own MCP servers.
- Search APIs take different approaches: some return ranked links and snippets, some are built for semantic/embedding-based retrieval, and some return LLM-generated answers with citations — alongside server-side search tools now built natively into the major model APIs.
- When passing scraped content into an agent's context, Markdown rather than raw HTML is the default choice across this whole tooling landscape, purely to keep token cost down.
- Fetched web content should be treated as untrusted input given prompt-injection risk, and an agent acting as a crawler should respect robots.txt and identify itself with a recognizable, contactable User-Agent.

## Three jobs — pick the right tool for each

| Job | What you need | Tools |
|---|---|---|
| 1. Read a known URL → clean text | URL-to-Markdown conversion | Jina Reader, Firecrawl /scrape, trafilatura, Mozilla Readability |
| 2. Crawl a site / scrape at scale | Multi-URL crawl + JS rendering + anti-bot | Firecrawl /crawl, Apify, Crawlee, Browserbase, Bright Data, ScrapingBee |
| 3. Search the web | Search query → ranked URLs + snippets or answers | Tavily, Exa, Brave Search API, Serper, Perplexity Sonar API, built-in provider tools |

## Job 1: Reader / URL-to-Markdown

**Jina Reader** (SaaS + OSS) — prefix any URL with `https://r.jina.ai/` and receive clean Markdown optimized for LLMs. No key required for basic usage; optional API key for higher rate limits. The extraction model is ReaderLM-v2 (1.5B). Supports PDF and MS Office documents via direct POST. Free tier available; OSS branch at github.com/jina-ai/reader. Exposes an MCP server via the Jina AI MCP (smithery.ai registry).

**Firecrawl /scrape** (SaaS + AGPL self-host) — one-URL scrape endpoint returning Markdown, HTML, or structured JSON. Handles JS-rendered pages, proxy rotation, and CAPTCHA. Free tier (500 credits/month); paid from $19/month. GitHub: github.com/mendableai/firecrawl. Exposes an official MCP server.

**Self-hosted options (OSS):**

- *trafilatura* (Apache 2.0, Python) — extracts main text and metadata from HTML with high accuracy; outputs TXT, Markdown, CSV, JSON, or TEI-XML. Command-line and library. Used by HuggingFace, IBM, and Microsoft Research. Docs: trafilatura.readthedocs.io.
- *Mozilla Readability* (Apache 2.0, JavaScript) — the parser behind Firefox Reader Mode; strips nav/ads and returns article DOM. GitHub: github.com/mozilla/readability. Pair with Playwright or Puppeteer for JS-rendered pages.

**When to self-host vs use a service:** self-hosted options are free and private but require infrastructure and cannot solve CAPTCHAs. Services handle anti-bot at scale out of the box.

## Job 2: Crawl / scrape at scale

**Firecrawl /crawl** (SaaS + AGPL self-host) — crawls an entire site and returns all pages as Markdown. Same service as the /scrape endpoint; the /crawl endpoint accepts a root URL and traverses all sub-URLs. Handles JS rendering, rate limiting, and proxy rotation automatically.

**Apify** (SaaS) — managed cloud platform with tens of thousands of community-built Actors (preconfigured scrapers for common targets; the store advertises 46,000+) plus a proxy network and storage layer. Actors run serverlessly; pricing is pay-per-compute-unit. Since June 2026, **20,000+ public Actors are also payable per run over x402** (USDC on Base, in partnership with Coinbase) — an agent can call an Actor with no Apify account and no API key; see /resources/agent-wallets-x402. Two Actors worth knowing by name: **Website Content Crawler** (site → clean, LLM-ready Markdown for RAG pipelines; Apify's June 2026 update announcement reports a faster, cheaper build with no config changes needed) and **AI Web Scraper**, whose **"Agentic" extraction mode** takes a plain-language prompt, reads the site's sitemap, plans its own crawl, and returns structured data — steerable with navigation hints in the prompt ("only follow product pages and pagination"), with no selectors to break on layout changes. Homepage: apify.com.

**Crawlee** (Apache 2.0, OSS by Apify) — open-source TypeScript/JavaScript (and Python) web-scraping library. Supports Cheerio, JSDOM, Playwright, and Puppeteer crawlers with auto proxy rotation, fingerprinting, and autoscaling. Can run locally or deploy to Apify. Python port stable since September 2025. GitHub: github.com/apify/crawlee.

**Browserbase** (SaaS) — managed cloud headless browsers (Playwright/Puppeteer API) optimized for AI agents. Handles CAPTCHA, stealth, and session recording. Priced per session. Homepage: browserbase.com.

**Bright Data** (SaaS) — enterprise proxy + scraping stack. Web MCP server (free tier: 5,000 requests/month) exposes Web Unlocker, SERP API, and Scraping Browser directly to MCP-compatible agents. Homepage: brightdata.com.

**ScrapingBee** (SaaS) — headless browser scraping API; handles JS rendering and proxy rotation. Acquired by Oxylabs in 2025; operates as an independent brand. Homepage: scrapingbee.com.

## Job 3: Search APIs

**Tavily** (SaaS) — agent-native search API: Search, Extract, Map, and Crawl endpoints. Returns structured results optimized for RAG. Sub-200ms p50 latency; 100M+ monthly requests. Joined Nebius (AI infrastructure) in February 2026. MCP server available. Docs: docs.tavily.com.

**Exa** (SaaS) — formerly Metaphor; neural/embedding-based search designed for AI agents. Retrieves pages by semantic meaning, not keyword matching. Raised $85M at $700M valuation (September 2025). APIs: Search, Contents, Answer, Find Similar, Websets. Contents (up to 10 results) included free with each Search call as of March 2026. Docs: exa.ai/docs.

**Brave Search API** (SaaS) — REST API over Brave's own independent web index (30B+ pages). Does not license from Google or Bing. SOC 2 Type II attested (October 2025). Supplies real-time search data to several major LLMs. Docs: brave.com/search/api.

**Serper** (SaaS) — fast Google SERP API. Returns real-time Google results (web, news, images, maps) in JSON. ~2.87s latency; $0.30–$1.00/1k queries at scale. 2,500 free queries/month. MCP server available. Homepage: serper.dev.

**Perplexity Sonar API** (SaaS) — LLM-generated answers with inline web citations. Four model tiers: Sonar, Sonar Pro, Sonar Reasoning, and Deep Research. $14–$22 per 1,000 Pro Search queries. Docs: docs.perplexity.ai.

**Built-in provider search tools** — all three major providers expose native web-search tools that run server-side (no extra API key needed):

- *Anthropic Claude* — `web_search_20260209` server tool in the Messages API; supports domain filtering, `max_uses` cap, and dynamic result filtering via code execution. $10 per 1,000 searches plus token costs. Docs: platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool.
- *OpenAI Responses API* — `{"type": "web_search"}` built-in tool; supports `external_web_access`, filters, and `return_token_budget` controls. Docs: platform.openai.com/docs/guides/tools-web-search.
- *Google Gemini* — `google_search` grounding tool; can be combined with custom function calling in a single API call. Docs: ai.google.dev/gemini-api/docs/google-search.

## The agent angle

Several services expose **MCP servers** (Jina, Firecrawl, Bright Data, Tavily, Serper), letting any MCP-compatible agent call web-data tools without custom integration. Check each provider's MCP docs or the registry at registry.modelcontextprotocol.io.

Clean **Markdown is the standard interchange** between web-data tools and agent context. Prefer it over raw HTML to minimize token cost.

When your agent is the crawler, respect robots.txt and AI-crawler policies: see /resources/ai-crawler-policy.

Treat all fetched web content as untrusted — prompt injection is a real attack surface. See /resources/agentic-security-checklist, sections 1 and 4.

Web data that feeds a retrieval system connects to the RAG layer: see /resources/rag-retrieval-for-agents.

## Practical guidance

- **Prefer reader/Markdown over raw HTML** — token cost difference is often 5–10x.
- **Cache aggressively** — a 1-hour TTL covers most agent use cases and cuts cost and latency substantially.
- **JS-rendered vs static** — static pages work with lightweight extractors (trafilatura, Readability). JS-heavy sites require a headless browser (Playwright, Browserbase, Bright Data Scraping Browser).
- **Rate-limit and identify your crawler honestly** — set a recognizable `User-Agent` with a contact URL; back off on 429; honor `Crawl-delay` in robots.txt.
- **For scale** — managed services (Firecrawl, Apify, Bright Data) handle proxy rotation and anti-bot. Self-hosted stacks (Crawlee + Playwright) give more control at higher ops cost.

## Verified sources

- Jina Reader API: https://jina.ai/reader/
- Jina Reader GitHub (jina-ai/reader): https://github.com/jina-ai/reader
- Firecrawl homepage: https://www.firecrawl.dev/
- Firecrawl GitHub (mendableai/firecrawl, AGPL-3.0): https://github.com/mendableai/firecrawl
- Crawlee GitHub (apify/crawlee, Apache 2.0): https://github.com/apify/crawlee
- Crawlee Python GitHub: https://github.com/apify/crawlee-python
- Apify platform: https://apify.com/
- Apify x402 launch — 20,000+ Actors payable per run (June 2026): https://blog.apify.com/introducing-x402-agentic-payments/
- Apify Website Content Crawler: https://apify.com/apify/website-content-crawler
- Apify AI Web Scraper (Agentic extraction mode): https://apify.com/apify/ai-web-scraper
- Trafilatura docs: https://trafilatura.readthedocs.io/
- Mozilla Readability GitHub: https://github.com/mozilla/readability
- Bright Data Web MCP blog: https://brightdata.com/blog/ai/web-scraping-with-mcp
- Tavily docs: https://docs.tavily.com/
- Exa Search API docs: https://exa.ai/docs/reference/search-api-guide
- Brave Search API: https://brave.com/search/api/
- Brave Search API growth announcement: https://brave.com/blog/search-api-growth/
- Perplexity Sonar API docs: https://docs.perplexity.ai/
- Anthropic web_search tool docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
- OpenAI web search (Responses API): https://platform.openai.com/docs/guides/tools-web-search
- Google Gemini grounding with Search: https://ai.google.dev/gemini-api/docs/google-search


Document Extraction and Parsing for Agents

Practitioner reference for the document-ingestion pipeline agents use: parse/OCR, layout/structure extraction, schema-constrained field extraction — with a verified tooling landscape (OSS and cloud).

Agents that ingest PDFs, scans, and tabular documents must solve three distinct problems before any LLM reasoning happens: (1) get readable text and structure out of the file, (2) preserve layout cues — tables, headings, reading order — so chunking is coherent, and (3) optionally extract specific fields into a schema. These are separate stages with separate failure modes.

## The two-stage distinction

**Stage A — Document-to-structure parsing:** Convert raw bytes (PDF, DOCX, image) into clean Markdown or structured JSON that an LLM can read cheaply. The output is a faithful representation of the document, not a business object.

**Stage B — Schema-constrained field extraction:** Take the structured output from Stage A (or feed the raw document to a VLM directly) and extract specific fields — invoice total, signatory name, table rows — into a validated schema. This is where structured-output guarantees from the LLM layer apply (see /resources/reliable-tool-calling).

Conflating A and B is the most common architecture mistake: trying to do field extraction before the document is cleanly parsed, or running a heavyweight extraction pipeline on documents that only need Markdown for RAG.

## Three parsing approaches

**1. Traditional OCR + layout analysis** — a pipeline of specialized models: text detection, character recognition, layout classification, table reconstruction, reading-order recovery. Battle-tested on high-volume document processing. Hard cases: multi-column PDFs, rotated scans, overlapping text/image regions, complex table spans. The cloud services below handle this tier.

**2. Vision-language models (VLMs) reading documents directly** — pass a page image to a multimodal LLM and ask it to return Markdown or JSON. Higher per-page cost than traditional OCR but handles layout ambiguity that rule-based pipelines miss. Best for low-to-medium volume, complex layouts, or when the extraction schema is known upfront.

**3. Document-parsing libraries and services targeting LLM output** — tools purpose-built to produce LLM-ready Markdown/JSON: they combine OCR, layout models, and optional VLM passes internally, exposing a clean API. The open-source tools below fall here.

## Open-source and library tooling

**Docling** (IBM, MIT license, github.com/docling-project/docling) — converts PDF, DOCX, PPTX, XLSX, HTML, EPUB, images, and more into Markdown or JSON. Includes TableFormer for table structure recovery, reading-order detection, and formula handling. Integrates with LangChain and LlamaIndex. Hosted under the LF AI & Data Foundation. Verified open-source.

**Marker** (Datalab / Vik Paruchuri, GPL-3.0 code / AI Pubs Open Rail-M weights, github.com/datalab-to/marker) — converts PDF, DOCX, PPTX, XLSX, EPUB, images to Markdown or JSON with optional JSON Schema extraction. Uses Surya internally for OCR and layout. Cloud API available at datalab.to. ~36K GitHub stars. Weights are free for research and startups under $2M revenue; commercial licensing required beyond that.

**Surya OCR** (Datalab, Apache 2.0 code / AI Pubs Open Rail-M weights, github.com/datalab-to/surya) — 650M-parameter model for OCR, layout analysis, reading-order detection, and table recognition in 90+ languages. Used as the backbone inside Marker. Weights free for research and startups under $5M revenue.

**Unstructured** (Unstructured-IO, Apache 2.0, github.com/Unstructured-IO/unstructured) — document ETL library for LLMs; partitions 40+ document types into typed elements (Title, NarrativeText, Table, Image), with 30+ connectors for data sources. Widely used for RAG ingestion pipelines. Also available as a hosted platform for production-grade workflows.

**MinerU** (OpenDataLab, custom Apache-2.0-based license since v3.1, github.com/opendatalab/MinerU) — high-accuracy PDF-to-Markdown/JSON engine supporting 109 languages via a VLM+OCR dual engine. Converts PDF, DOCX, PPTX, XLSX, images, and web pages; preserves tables as HTML, formulas as LaTeX. ~68K GitHub stars.

**LlamaParse** (LlamaIndex, cloud SaaS, llamaindex.ai/llamaparse) — managed document parsing service; four processing tiers (Fast, Cost Effective, Agentic, Agentic Plus); outputs LLM-ready Markdown and structured JSON; free tier includes ~1,000 pages/month. Backed by GPT-4.1 and Gemini 2.5 Pro for complex layouts (as of May 2025).

## Cloud / managed services

**AWS Textract** (Amazon, SaaS, docs.aws.amazon.com/textract/) — layout-aware OCR returning blocks, key-value pairs, tables, and query responses as structured JSON. Synchronous API for images; asynchronous batch for multi-page PDFs. Native fit for AWS-first teams; tight S3 / Lambda integration.

**Azure AI Document Intelligence** (Microsoft, SaaS, learn.microsoft.com/azure/ai-services/document-intelligence/) — prebuilt models for invoices, receipts, tax forms, IDs, and general layout; custom models for domain-specific extraction. REST and SDK (Python, C#, Java, JS). GA version 4.0 (API 2024-11-30). Strong choice for Microsoft-stack environments.

**Google Document AI** (Google Cloud, SaaS, cloud.google.com/document-ai) — layout-aware OCR with pretrained processors for forms, invoices, and identity documents; custom extractor backed by Gemini 2.5 Pro (Preview, June 2025). Returns structured JSON including key-value pairs, tables, and bounding-box coordinates.

**Mistral OCR** (Mistral AI, SaaS API, docs.mistral.ai/models/model-cards/ocr-4-0) — cloud OCR API accessed via the `/v1/ocr` endpoint or SDK. Latest version: OCR 4 (shipped June 23, 2026), superseding OCR 3 (v25.12, December 2025), which remains available. Returns Markdown with HTML table reconstruction; handles interleaved images, math, and complex layouts; OCR 4 adds paragraph-level bounding boxes, typed-block classification (titles, tables, equations, signatures, etc.), inline confidence scores, and coverage of 170 languages across 10 language groups. Available via the Mistral API, Document AI in Mistral Studio, AWS SageMaker, and Microsoft Foundry. Mistral also reportedly offers OCR 4 as a self-hosted single-container Docker deployment for an organization's own infrastructure (enterprise-only, data-residency/compliance use case, no document data leaves the org's environment) — reported by outlets that led their coverage with the self-hosting angle (techtimes.com, digitalmatters.me, aiweekly.co); not independently primary-fetched this session (mistral.ai and docs.mistral.ai fetches were blocked), so re-verify against mistral.ai/news/ocr-4/ before relying on it. Pricing: $4 per 1,000 pages standard API (up from $2 for OCR 3), with the same 50%-off Batch API discount ratio, i.e. $2 per 1,000 pages batch (up from $1) — the price roughly doubled in both tiers, not just one. A separate Mistral-owned product, Document AI (schema-driven extraction inside Mistral Studio, not to be confused with **Google Document AI** above), is priced at $5 per 1,000 pages. Verified SaaS; model weights are not open. Facts in this bullet are WebSearch-corroborated across independent secondary outlets, not primary-fetched this session.

**Reducto** (Reducto AI, SaaS, reducto.ai) — agentic document platform providing layout-aware OCR, parse/split/extract endpoints, and schema-grounded field extraction for production agent pipelines. 1B+ pages processed; targets enterprise accuracy requirements. Paid; funding verified ($108M Series B).

## Structured field extraction (Stage B)

Once a document is parsed to Markdown or JSON, field extraction is a structured-output problem: define a JSON Schema for the target fields, run an LLM with constrained decoding or tool-calling strict mode, and validate the result. See /resources/reliable-tool-calling for the full pattern.

VLM-based extraction (pass the page image directly to a multimodal model with a schema prompt) skips Stage A but is costlier per page and harder to debug when fields are missed.

**Hard cases to test:** multi-column layouts, merged table cells, rotated or skewed scans, handwritten annotations, and forms with no consistent field label placement.

For high-stakes fields (financial, medical, legal), cross-validate extracted values against OCR confidence scores and route low-confidence results to a human review queue.

## The agent integration angle

Parsed document output (Markdown or JSON) feeds directly into the RAG layer. Clean structure at parse time is the single biggest lever on retrieval quality. See /resources/rag-retrieval-for-agents for chunking strategies and embedding choices; and /resources/embeddings-vector-search for index selection.

Document content is an untrusted surface: a malicious PDF can embed prompt-injection payloads in OCR-readable text. Strip or escape instruction-like patterns from parsed output before inserting it into agent context. See /resources/agentic-security-checklist, section 4 (untrusted content handling).

## Verified sources

- Docling (IBM, MIT): https://github.com/docling-project/docling
- Marker (Datalab, GPL-3.0): https://github.com/datalab-to/marker
- Surya OCR (Datalab, Apache 2.0): https://github.com/datalab-to/surya
- Unstructured (Apache 2.0): https://github.com/Unstructured-IO/unstructured
- MinerU (OpenDataLab): https://github.com/opendatalab/MinerU
- LlamaParse docs: https://developers.llamaindex.ai/python/cloud/llamaparse/tiers/
- AWS Textract docs: https://docs.aws.amazon.com/textract/latest/dg/what-is.html
- Azure AI Document Intelligence overview: https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview
- Google Document AI overview: https://cloud.google.com/document-ai/docs/overview
- Mistral OCR 4 model card (WebSearch-corroborated, not primary-fetched this session): https://docs.mistral.ai/models/model-cards/ocr-4-0
- Mistral — "Introducing OCR 4" news post (WebSearch-corroborated, not primary-fetched): https://mistral.ai/news/ocr-4/
- Mistral OCR 3 docs (prior version, kept for reference): https://docs.mistral.ai/models/ocr-3-25-12
- Reducto agentic document platform: https://reducto.ai/