Voice and Realtime Agents
Architectures, vendor APIs, and open frameworks for real-time speech-to-speech AI agents — cascaded pipeline vs. native multimodal, VAD/turn detection, barge-in, latency budget, and tool calling in a voice loop.
Real-time voice agents are one of the fastest-growing deployment patterns in 2026. Two architectures dominate. Understanding the tradeoffs between them is the prerequisite for every vendor and framework choice downstream.
Key facts
- Real-time voice agents split into two dominant architectures: a cascaded STT → LLM → TTS pipeline, and a single native speech-to-speech model that ingests and emits raw audio directly.
- The cascaded pipeline gives full control to swap any component — including running each stage on an open-weight model — but pays a latency and cost penalty from three sequential model calls per turn.
- Native speech-to-speech models offer lower end-to-end latency and richer built-in prosody/interruption handling, at the cost of vendor lock-in on the underlying model and expensive audio tokens; open-weight native speech-to-speech options are still limited as of mid-2026.
- Turn detection and barge-in/interruption handling are explicit engineering concerns in a cascaded pipeline but are handled inside the model for native speech-to-speech systems.
- Vendor realtime APIs from OpenAI, Google, Amazon, and xAI all now support mid-conversation tool/function calling alongside streaming audio.
- Transport choice tends to follow the use case — WebRTC for browser/mobile clients, WebSocket for server-to-server integration — with some vendors adding a SIP path for telephony.
- Some vendors design for drop-in compatibility with each other: xAI's realtime API matches the OpenAI Realtime API's specification, so existing OpenAI-built clients can point at it with minimal changes.
Architecture 1: Cascaded pipeline (STT → LLM → TTS)
The classic pipeline chains three separate models:
- STT — streaming speech-to-text converts the user's audio to a text transcript.
- LLM — the transcript is fed to a language model, which produces a text reply (and may call tools).
- TTS — the reply is synthesized back to audio.
A VAD (Voice Activity Detection) module sits upstream to detect when the user is speaking and trigger end-of-turn detection — the decision that the user has finished and the agent should respond. Between the models, barge-in / interruption handling flushes the TTS buffer and restarts the STT stage when the user speaks over the agent.
Tradeoffs:
| Dimension | Cascaded pipeline |
|---|---|
| Latency | Higher (three sequential models); target sub-second requires fast STT + cached LLM prefix + streaming TTS |
| Interruptibility | Requires explicit barge-in logic at each stage boundary |
| Emotion / prosody | TTS adds prosody; quality varies by provider |
| Cost | Pay for three separate model calls per turn |
| Control | High: swap any component independently; use any LLM |
| Open-weight path | Yes — each stage can run on open-weight models |
Architecture 2: Native speech-to-speech (realtime multimodal models)
A single model ingests raw audio and outputs raw audio directly, without a text intermediate at the core inference step. Turn detection, interruption handling, and prosody are handled inside the model.
Tradeoffs:
| Dimension | Native speech-to-speech |
|---|---|
| Latency | Lower end-to-end (one model, streaming output) |
| Interruptibility | Built into the model; lower barge-in latency |
| Emotion / prosody | Richer; the model controls vocal tone end-to-end |
| Cost | Single model call, but audio tokens are expensive |
| Control | Lower: you cannot swap the underlying LLM independently |
| Open-weight path | Limited — open-weight native speech-to-speech models are still emerging as of mid-2026 |
Vendor realtime APIs
All entries below are web-verified as of 2026-07-09.
OpenAI Realtime API
A native speech-to-speech API. The original gpt-realtime (GA August 28, 2025) was superseded by gpt-realtime-2 (May 7, 2026 — GPT-5-class reasoning, configurable reasoning effort, 128K context), then by gpt-realtime-2.1 and mini variant gpt-realtime-2.1-mini (July 6, 2026 — improved alphanumeric recognition, noise/silence handling, and interruption behavior; ~25% lower p95 latency via caching). List pricing for 2.1 is unchanged from gpt-realtime-2 / the original mini (audio ≈$32/$64 per 1M input/output tokens full model, ≈$10/$20 mini) — no price cut shipped alongside 2.1, despite some chatter to that effect. The earlier gpt-4o-realtime-preview series is deprecated.
Transports: WebRTC (recommended for browsers and mobile — lower jitter, handles NAT traversal) and WebSocket (recommended for server-to-server). A SIP integration path is also available for telephony.
Supports streaming audio input and output, tool/function calling mid-conversation, VAD and server-side turn detection, and barge-in. Approximate glass-to-glass latency: 300–600 ms on subsequent turns.
Docs: platform.openai.com/docs/guides/realtime-webrtc, developers.openai.com/api/docs/guides/realtime-websocket, and developers.openai.com/api/docs/models/gpt-realtime-2.1
Google Gemini Live API
A native speech-to-speech API with bidirectional streaming over WebSocket. The model processes audio input and returns audio output natively, without a text intermediate. GA model: Gemini 2.5 Flash (native audio), available via both Google AI for Developers and Vertex AI. A newer Gemini 3.1 Flash Live (preview, released March 26, 2026) adds sharper acoustic-nuance detection and lower latency, but as of this writing is available only via Google AI Studio — no Vertex AI availability or GA date has been announced.
Supports multimodal input (audio + video/screen), turn detection, barge-in, and function calling.
Docs: ai.google.dev/gemini-api/docs/live-api
Amazon Nova Sonic
A native speech-to-speech model on Amazon Bedrock, announced April 2025. The current generation is Amazon Nova 2 Sonic (December 2025). Accessed via Bedrock's bidirectional streaming API (WebSocket). Also supports WebRTC via an AWS blog reference implementation.
Supports tool use, voice selection, interruption handling, and background-noise robustness. Integrates with Amazon Connect and telephony providers (Vonage, Twilio) and open frameworks including LiveKit and Pipecat.
Docs: docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html
xAI Grok Voice Agent API
A realtime speech-to-speech API launched December 17, 2025. Uses bidirectional WebSocket streaming. Compatible with the OpenAI Realtime API specification, so clients built for OpenAI Realtime can point at xAI with minimal changes.
Features: custom VAD, Smart Turn end-of-turn detection, sub-1-second time-to-first-audio, 100+ language support with automatic detection. Also available via a native LiveKit plugin.
Docs: docs.x.ai/docs/guides/voice
Open frameworks and orchestrators
Pipecat (pipecat-ai)
Open-source Python framework (BSD-2-Clause) for building real-time voice and multimodal conversational agents, developed by Daily. Organizes processing as pipeline frames flowing through transport, STT, LLM, and TTS stages. Supports 20+ STT providers and 30+ TTS providers, plus direct integrations with native speech-to-speech services (OpenAI Realtime, Amazon Nova Sonic, Gemini Live).
Transports: WebRTC (Daily, LiveKit, SmallWebRTC), WebSocket, telephony. Handles VAD, turn detection, barge-in, and multi-agent coordination.
GitHub: github.com/pipecat-ai/pipecat
LiveKit Agents
Open-source Python and TypeScript framework (Apache 2.0) for building realtime voice, video, and physical AI agents on top of the LiveKit WebRTC infrastructure. SDK v1.0 GA April 2025; current release v1.8.3 (Sep 23 2026, confirmed via PyPI this session), up from the v1.6.x line cited in the prior pass.
The 1.0 release replaced the older VoicePipelineAgent and MultimodalAgent classes (both now deprecated) with a single unified orchestrator, AgentSession, which covers cascaded (STT → LLM → TTS) and native speech-to-speech backends (e.g. OpenAI Realtime, Gemini Live) without changing application code when switching between them. Includes built-in turn detection, barge-in, native MCP tool support, and function calling. Bring-your-own STT, LLM, and TTS with no lock-in.
Docs: docs.livekit.io/agents
STT and TTS component vendors
STT:
Deepgram — streaming STT via WebSocket. Nova-3 is the flagship model (low WER). Flux is a conversational STT model with model-integrated end-of-turn detection and configurable turn-taking dynamics, designed specifically for voice agent pipelines. Docs: developers.deepgram.com
whisper.cpp (ggml-org) — C/C++ port of OpenAI's Whisper ASR models; runs locally with no external dependencies. Supports GPU acceleration and VAD. Use for on-device or self-hosted STT when latency from network round-trips to a cloud STT API is a constraint. GitHub: github.com/ggml-org/whisper.cpp
TTS:
ElevenLabs — streaming TTS via WebSocket. Eleven v3 model. Broadest language coverage (70+ languages), strong voice cloning. Streaming endpoint:
/v1/text-to-speech/{voice_id}/stream-input. Docs: elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-inputCartesia — streaming TTS via WebSocket. Sonic-3.5 (released June 2026) is the current flagship — approx. 40 ms time-to-first-audio on the Turbo variant, sub-100 ms on standard, native support for 42 languages. The prior Sonic 3 model remains available. Optimized for streaming latency at scale. Docs: docs.cartesia.ai
Key concepts
- VAD (Voice Activity Detection) — a classifier that scores incoming audio for voice presence, typically using an energy threshold plus a voice classifier plus a minimum-duration guard. Determines when the user is speaking.
- End-of-turn detection / endpointing — deciding that the user has finished their utterance and the agent should respond. Can be energy-based (silence duration), STT-integrated (model signals completion), or model-based (an ML classifier predicts semantic completion rather than just silence). Model-based detection has lower latency than waiting for silence timeouts.
- Barge-in / interruption handling — detecting when the user speaks while the agent is responding, flushing the TTS output buffer, and restarting the listen/respond cycle. Failure to handle barge-in correctly is the most common CSAT failure in voice agents — distinguish genuine interruptions from backchannels ("uh-huh", "right") to avoid cutting off mid-sentence unnecessarily.
- Latency budget — target sub-600 ms glass-to-glass for natural-feeling turn-taking. In a cascaded pipeline, the budget is roughly: STT (50–150 ms) + LLM time-to-first-token (100–300 ms) + TTS time-to-first-audio (50–150 ms). Each component must be streaming — do not wait for full STT transcript before starting LLM inference. See /resources/agent-cost-latency-optimization.
- Tool calling in a voice loop — voice agents can call tools mid-conversation, but tool latency adds directly to voice latency. Keep tool calls under 200 ms; use speculative execution for predictable tool calls; return partial results to the model via streaming where possible. See /resources/reliable-tool-calling.
Practical guidance
- Measure end-to-end latency per component (STT → LLM → TTS), not just total. Instrument each stage separately; bottlenecks are rarely where intuition points.
- Handle interruptions before tuning latency — a fast agent that cannot be interrupted is worse than a slightly slower one that can.
- Keep tool calls fast — tool latency is voice latency. Parallelize independent tool calls; cache results of stable lookups.
- Plan for transcription errors — cascade pipelines inherit STT errors. Design prompts and tool schemas to tolerate common transcription noise (homophones, dropped words).
- Budget for always-on audio cost — native speech-to-speech APIs price audio tokens at rates 3–10× higher than text tokens. Profile real traffic before committing to a pricing model.
- For latency optimization across the full agent stack, see /resources/agent-cost-latency-optimization.
- For tool-calling reliability in a voice loop, see /resources/reliable-tool-calling.
- For framework choices when adding voice to a multi-agent system, see /resources/agent-frameworks-compared.
Verified sources
Re-verified 2026-09-28 (M221 corpus pass #14): LiveKit Agents' version claim was checked directly against its current PyPI release (livekit-agents v1.8.3, Sep 23 2026) and updated above (was v1.6.x). Pipecat and whisper.cpp were re-confirmed active/reachable directly (pipecat-ai v1.12.0 on PyPI, Sep 26 2026; the whisper.cpp GitHub repo still resolves). The vendor realtime-API section below — OpenAI, Google, Amazon, xAI, Deepgram, ElevenLabs, Cartesia — could not be independently re-verified this session: every one of those vendor doc/blog domains (platform.openai.com, developers.openai.com, community.openai.com, ai.google.dev, docs.aws.amazon.com, docs.x.ai, developers.deepgram.com, elevenlabs.io, docs.cartesia.ai, marktechpost.com) refused the connection under this session's organization egress policy, and no WebSearch/WebFetch tool was available as a fallback. Those facts (model names, pricing, latency figures, release dates) are carried forward unchanged from the prior 2026-07-09 verification, not re-confirmed this cycle — flagged explicitly per M221's instructions rather than force-bumped.
- OpenAI Realtime API (WebRTC): https://platform.openai.com/docs/guides/realtime-webrtc
- OpenAI Realtime API (WebSocket): https://developers.openai.com/api/docs/guides/realtime-websocket
- OpenAI gpt-realtime GA announcement: https://openai.com/index/introducing-gpt-realtime/
- OpenAI gpt-realtime-2.1 / gpt-realtime-2.1-mini announcement (July 6, 2026): https://community.openai.com/t/new-realtime-models-on-the-api-gpt-realtime-2-1-and-gpt-realtime-2-1-mini/1385896
- gpt-realtime-2.1 pricing unchanged from gpt-realtime-2/original mini (no cut): https://www.marktechpost.com/2026/07/06/openai-gpt-realtime-2-1-mini-reasoning-realtime-api/
- Google Gemini Live API overview: https://ai.google.dev/gemini-api/docs/live-api
- Google Gemini 3.1 Flash Live (preview, March 26, 2026): https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview
- Google Gemini Live API (WebSocket get started): https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket
- Amazon Nova Sonic announcement (April 2025): https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-sonic-speech-to-speech-conversations-bedrock/
- Amazon Nova 2 Sonic announcement (December 2025): https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-nova-2-sonic-real-time-conversational-ai
- Amazon Nova Sonic bidirectional streaming docs: https://docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html
- xAI Grok Voice Agent API announcement: https://x.ai/news/grok-voice-agent-api
- xAI Voice Agent API docs: https://docs.x.ai/docs/guides/voice
- Pipecat (pipecat-ai, BSD-2-Clause): https://github.com/pipecat-ai/pipecat
- LiveKit Agents docs: https://docs.livekit.io/agents/voice-agent/
- LiveKit Agents source (v1.6.x, Apache-2.0, confirms AgentSession as the current API): https://github.com/livekit/agents
- Deepgram Nova-3 and Flux STT docs: https://developers.deepgram.com/docs/models-languages-overview
- whisper.cpp (ggml-org, C/C++ Whisper port): https://github.com/ggml-org/whisper.cpp
- ElevenLabs TTS WebSocket API: https://elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-input
- Cartesia TTS WebSocket API: https://docs.cartesia.ai/api-reference/tts/websocket
- Cartesia Sonic-3.5 / Ink-2 launch (June 2026): https://cartesia.ai/launch
Free to read, always. Want this whole reference corpus inside your own agents? €5 unlocks every premium reference for one agent; €25 licenses the full corpus as RAG / fine-tuning data with an AI-use grant (procurement one-pager: /corpus-license); €150 adds redistribution rights.