AI Gateways and LLM Routing
What an AI gateway is, routing strategies (failover, cost-cascade, latency, capability), the tooling landscape, the OpenAI-compatible API convention, and tradeoffs.
An AI gateway (also called an LLM proxy or LLM router) is a single reverse-proxy that sits in front of multiple model providers and exposes one unified endpoint to callers. Instead of coding provider-specific clients for OpenAI, Anthropic, Google, Bedrock, and others, every agent sends requests to the gateway; the gateway routes, retries, caches, rate-limits, and logs on behalf of the caller.
For production agents, a gateway solves five categories of problems simultaneously: provider reliability (failover and retries), cost control (budget caps, model cascades), performance (caching, load balancing), security (key management, PII redaction), and observability (centralized logging of every request — see /resources/agent-observability).
Key facts
- An AI gateway is a single reverse-proxy standing between an agent and several different model providers at once, giving agents one unified endpoint instead of provider-specific client code.
- A gateway tackles five problem areas for production agents at once: provider reliability, cost control, performance, security, and observability.
- Routing strategies span failover, latency-based, cost-based/model-cascade, and capability-based approaches, each suited to a different tradeoff.
- Most gateways speak the de-facto OpenAI-compatible chat-completions request/response shape, which is what lets an agent swap its upstream provider with no code change.
- The tooling landscape splits between self-hosted, open-source options (LiteLLM, Portkey, Kong) and hosted SaaS aggregators (OpenRouter, Cloudflare AI Gateway, Helicone).
- Tradeoffs include the added network hop's latency, a new single point of failure, and data-privacy exposure since every prompt and response passes through the gateway operator.
- Self-hosting trades operational effort for control, while SaaS gateways stay lightweight to run but create a dependency on the vendor.
Core gateway capabilities
- Unified endpoint — one
POST /v1/chat/completions-compatible URL in front of many providers. - Provider failover — if provider A returns 5xx or times out, retry on provider B automatically.
- Load balancing — distribute traffic across multiple deployments of the same model.
- Caching — exact-match response caching for repeated prompts; semantic caching for near-duplicate queries using embedding similarity (see /resources/agent-memory-context).
- Rate limiting and budget controls — per-key, per-team, or per-model spend caps and request-per-minute limits enforced before upstream charges accrue.
- Key management — virtual keys issued to callers; the gateway holds the real provider keys.
- Observability — every request logged with latency, cost, tokens, and model used (see /resources/agent-observability).
Routing strategies
Provider failover — primary provider first; on error (5xx, timeout) automatically retry on one or more fallback providers. Protects uptime without code changes in the agent.
Latency-based routing — measure provider response time and weight traffic toward the fastest responder. Useful when several providers offer the same model (e.g., via OpenRouter).
Cost-based / model cascade — route the request to the cheapest model capable of handling it. A classifier or confidence score determines whether to escalate to a more capable (and expensive) model. Also called "mixture-of-agents" or "model cascade." RouteLLM (see below) implements this as a standalone router.
Capability routing — send image inputs to a vision-capable model, audio to a speech-capable model, and code to a code-specialized model, based on detected input type or an explicit hint in the request.
The OpenAI-compatible API convention
Most gateways and aggregators expose the POST /v1/chat/completions endpoint shape (request: model, messages, temperature; response: choices[].message). This is a de-facto industry standard, not a ratified spec — it originated with OpenAI and was copied by most providers. The newer POST /v1/responses shape (OpenAI Responses API, released March 2025) is increasingly supported but adoption is not yet universal. Because clients target this shape, swapping the gateway's upstream provider requires no code change on the agent side.
Tooling landscape
LiteLLM — open-source proxy (MIT license; the enterprise/ directory ships under a separate commercial license) that added a Rust core alongside its Python SDK; its own GitHub tagline now reads "the fastest, litest AI Gateway... Call 100+ LLM APIs" (as of July 2026), a headline-number and architecture shift from the project's earlier "140+ providers / 2,600+ models" framing — the underlying provider table still lists well over 100 distinct providers. Features: virtual keys, budget caps, fallbacks, load balancing, semantic caching, logging integrations. 54k+ GitHub stars as of July 2026. Self-host via Docker; enterprise SaaS tier available. Security note: a critical pre-auth SQL injection in the proxy (CVE-2026-42208, CVSS 9.3) was disclosed in April 2026 and exploited within days — fixed in v1.83.7+; pin to a patched release. Source: github.com/BerriAI/litellm.
OpenRouter — hosted SaaS aggregator (no self-host option). One OpenAI-compatible API reaching 400+ models from 70+ providers. Per-token rates match provider-published prices (no per-token markup), but a platform fee (~5.5%, minimum $0.80) applies on credit purchases. Includes an Auto Router (powered by NotDiamond) that selects the best model per prompt automatically. Useful for latency-based and cost-based routing without running infrastructure. Source: openrouter.ai/docs.
Portkey — open-source AI gateway (Apache 2.0, gateway 2.0 fully open-sourced March 2026; github.com/Portkey-AI/gateway) with a managed SaaS cloud option. Routes to 250+ providers; includes semantic caching, guardrails, RBAC, observability, and MCP gateway support. Acquired by Palo Alto Networks (completed May 29, 2026); Portkey's gateway now serves as the AI Gateway layer for Palo Alto's Prisma AIRS platform. The open-source routing engine remains on GitHub under Apache 2.0; managed-tier features (observability storage, semantic caching at scale, team RBAC) stay cloud-only. Source: portkey.ai/docs.
Cloudflare AI Gateway — hosted SaaS (no self-host; Cloudflare-infrastructure only). Free tier available. One URL change routes any supported provider through Cloudflare's edge. Features: caching, rate limiting, logging, guardrails, request retry/fallback. Added a unified REST API (api.cloudflare.com) in May 2026. Source: developers.cloudflare.com/ai-gateway.
Helicone — open-source LLM observability platform and AI gateway (github.com/Helicone/helicone; YC W23). Integration model: change one URL. Provides request logging, cost tracking, caching, rate limiting, and agent trace inspection. Self-host via Docker/Helm or use the SaaS cloud (10k requests/month free). Acquired by Mintlify (March 3, 2026); now in maintenance mode — security patches, bug fixes, and new-model support only, no active feature development.
Kong AI Gateway — AI routing and governance layer built on top of Kong Gateway (the established API gateway). Adds AI-specific plugins: AI Proxy, AI Proxy Advanced (multi-provider load balancing), PII redaction, allow/deny lists for prompts. Kong AI Gateway 3.14 (April 2026) added "Kong Agent Gateway," making Agent-to-Agent (A2A) governance generally available alongside the LLM and MCP layers. Open-source core (Apache 2.0); enterprise tier available. Source: developer.konghq.com/ai-gateway.
RouteLLM — open-source model router framework (Apache 2.0; github.com/lm-sys/routellm; by LMSYS / Anyscale). Trains classifiers on human preference data to predict which model will produce better output. Routes simpler queries to cheap models, complex ones to capable models. Exposes an OpenAI-compatible server. Not a full gateway (no caching, key management, etc.) — a focused cost-based routing layer.
Tradeoffs
- Added latency — the gateway is an extra network hop; well-operated gateways add <10 ms but under load this can grow.
- New failure point — if the gateway is down, all model calls fail. Self-hosted gateways require you to operate and scale them; SaaS gateways transfer that risk to the vendor.
- Data privacy — all prompts and responses pass through the gateway operator's infrastructure. For sensitive workloads, prefer self-hosted gateways or providers with strong data-processing agreements.
- Vendor lock-in vs self-hosting — SaaS gateways are operationally lightweight but create a dependency; self-hosted gateways (LiteLLM, Portkey, Kong) give full control at the cost of infrastructure work.
Cross-links: gateway logs are a primary telemetry source for agent observability (/resources/agent-observability); semantic caching at the gateway layer is a form of context reuse (/resources/agent-memory-context); routing to self-hosted models is a key gateway use case (/resources/open-weight-models-for-agents).
Verified sources
- LiteLLM GitHub (BerriAI): https://github.com/BerriAI/litellm
- LiteLLM proxy docs: https://docs.litellm.ai/docs/simple_proxy
- OpenRouter docs: https://openrouter.ai/docs
- OpenRouter Auto Router docs: https://openrouter.ai/docs/guides/routing/routers/auto-router
- Portkey gateway GitHub (Portkey-AI): https://github.com/Portkey-AI/gateway
- Portkey docs: https://portkey.ai/docs
- Cloudflare AI Gateway overview: https://developers.cloudflare.com/ai-gateway/
- Cloudflare AI Gateway REST API changelog (May 2026): https://developers.cloudflare.com/changelog/post/2026-05-21-rest-api/
- LiteLLM CVE-2026-42208 advisory: https://docs.litellm.ai/blog/cve-2026-42208-litellm-proxy-sql-injection
- Palo Alto Networks completes Portkey acquisition (May 2026): https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-completes-acquisition-of-portkey-to-secure-ai-agents
- Helicone GitHub: https://github.com/Helicone/helicone
- Helicone joins Mintlify (March 2026): https://www.helicone.ai/blog/joining-mintlify
- Kong AI Gateway docs: https://developer.konghq.com/ai-gateway/
- RouteLLM GitHub (lm-sys): https://github.com/lm-sys/routellm