{
  "slug": "voice-realtime-agents",
  "title": "Voice and Realtime Agents",
  "description": "Architectures, vendor APIs, and open frameworks for real-time speech-to-speech AI agents — cascaded pipeline vs. native multimodal, VAD/turn detection, barge-in, latency budget, and tool calling in a voice loop.",
  "category": "Guide",
  "tags": [
    "voice",
    "realtime",
    "speech",
    "stt",
    "tts",
    "vad",
    "agents",
    "webrtc",
    "websocket"
  ],
  "updated": "2026-07-09",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/voice-realtime-agents",
  "markdown": "https://changegamer.ai/resources/voice-realtime-agents.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "Architecture 1: Cascaded pipeline (STT → LLM → TTS)",
      "anchor": "architecture-1-cascaded-pipeline-stt-llm-tts"
    },
    {
      "depth": 2,
      "text": "Architecture 2: Native speech-to-speech (realtime multimodal models)",
      "anchor": "architecture-2-native-speech-to-speech-realtime-multimodal-models"
    },
    {
      "depth": 2,
      "text": "Vendor realtime APIs",
      "anchor": "vendor-realtime-apis"
    },
    {
      "depth": 3,
      "text": "OpenAI Realtime API",
      "anchor": "openai-realtime-api"
    },
    {
      "depth": 3,
      "text": "Google Gemini Live API",
      "anchor": "google-gemini-live-api"
    },
    {
      "depth": 3,
      "text": "Amazon Nova Sonic",
      "anchor": "amazon-nova-sonic"
    },
    {
      "depth": 3,
      "text": "xAI Grok Voice Agent API",
      "anchor": "xai-grok-voice-agent-api"
    },
    {
      "depth": 2,
      "text": "Open frameworks and orchestrators",
      "anchor": "open-frameworks-and-orchestrators"
    },
    {
      "depth": 3,
      "text": "Pipecat (pipecat-ai)",
      "anchor": "pipecat-pipecat-ai"
    },
    {
      "depth": 3,
      "text": "LiveKit Agents",
      "anchor": "livekit-agents"
    },
    {
      "depth": 2,
      "text": "STT and TTS component vendors",
      "anchor": "stt-and-tts-component-vendors"
    },
    {
      "depth": 2,
      "text": "Key concepts",
      "anchor": "key-concepts"
    },
    {
      "depth": 2,
      "text": "Practical guidance",
      "anchor": "practical-guidance"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "agent-cost-latency-optimization",
      "title": "Agent Cost and Latency Optimization",
      "description": "Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).",
      "url": "https://changegamer.ai/resources/agent-cost-latency-optimization"
    },
    {
      "slug": "agent-identity-authentication",
      "title": "Agent Identity and Authentication",
      "description": "How autonomous agents prove who they are and get authorized to act: workload identity vs. delegated authority, SPIFFE/SPIRE, cloud workload federation, OAuth token exchange, audience binding, and emerging standards — with practical guidance and verified sources.",
      "url": "https://changegamer.ai/resources/agent-identity-authentication"
    },
    {
      "slug": "agent-memory-context",
      "title": "Agent Memory and Context Management",
      "description": "Architecture reference for agent memory: types (working, long-term, episodic, semantic, procedural), context-management techniques (summarization, RAG, sliding windows, prompt caching), storage substrates, and memory frameworks — with security notes and cross-links to related guides.",
      "url": "https://changegamer.ai/resources/agent-memory-context"
    },
    {
      "slug": "agent-observability",
      "title": "Agent Observability and Tracing",
      "description": "Why agents need observability beyond app logs, how OpenTelemetry GenAI semantic conventions model agent runs as traces, key signals to capture, and a verified tooling landscape.",
      "url": "https://changegamer.ai/resources/agent-observability"
    }
  ],
  "furtherReading": [
    {
      "slug": "acp-vs-ap2-vs-x402",
      "title": "ACP vs. AP2 vs. x402: Which Agent Payment Rail Should You Implement?",
      "description": "A decision framework for choosing between ACP, AP2, and x402 (plus the self-hosted 402 gate) — sorted by who your buyer actually is, what you are selling, and what is live versus waitlisted today.",
      "url": "https://changegamer.ai/articles/acp-vs-ap2-vs-x402"
    },
    {
      "slug": "agent-checkout-vs-human-checkout",
      "title": "Agent Checkout vs. Human Checkout: Why Your Payment Flow Fails Machine Buyers",
      "description": "Why checkout built for a person watching a screen is unusable by an AI agent, and what a checkout flow that actually completes for a machine buyer looks like — 402 + API key versus native x402.",
      "url": "https://changegamer.ai/articles/agent-checkout-vs-human-checkout"
    }
  ],
  "body": "Real-time voice agents are one of the fastest-growing deployment patterns in 2026. Two architectures dominate. Understanding the tradeoffs between them is the prerequisite for every vendor and framework choice downstream.\n\n## Key facts\n\n- Real-time voice agents split into two dominant architectures: a cascaded STT → LLM → TTS pipeline, and a single native speech-to-speech model that ingests and emits raw audio directly.\n- The cascaded pipeline gives full control to swap any component — including running each stage on an open-weight model — but pays a latency and cost penalty from three sequential model calls per turn.\n- Native speech-to-speech models offer lower end-to-end latency and richer built-in prosody/interruption handling, at the cost of vendor lock-in on the underlying model and expensive audio tokens; open-weight native speech-to-speech options are still limited as of mid-2026.\n- Turn detection and barge-in/interruption handling are explicit engineering concerns in a cascaded pipeline but are handled inside the model for native speech-to-speech systems.\n- Vendor realtime APIs from OpenAI, Google, Amazon, and xAI all now support mid-conversation tool/function calling alongside streaming audio.\n- Transport choice tends to follow the use case — WebRTC for browser/mobile clients, WebSocket for server-to-server integration — with some vendors adding a SIP path for telephony.\n- Some vendors design for drop-in compatibility with each other: xAI's realtime API matches the OpenAI Realtime API's specification, so existing OpenAI-built clients can point at it with minimal changes.\n\n## Architecture 1: Cascaded pipeline (STT → LLM → TTS)\n\nThe classic pipeline chains three separate models:\n\n1. **STT** — streaming speech-to-text converts the user's audio to a text transcript.\n2. **LLM** — the transcript is fed to a language model, which produces a text reply (and may call tools).\n3. **TTS** — the reply is synthesized back to audio.\n\nA **VAD (Voice Activity Detection)** module sits upstream to detect when the user is speaking and trigger **end-of-turn detection** — the decision that the user has finished and the agent should respond. Between the models, **barge-in / interruption handling** flushes the TTS buffer and restarts the STT stage when the user speaks over the agent.\n\n**Tradeoffs:**\n\n| Dimension | Cascaded pipeline |\n|---|---|\n| Latency | Higher (three sequential models); target sub-second requires fast STT + cached LLM prefix + streaming TTS |\n| Interruptibility | Requires explicit barge-in logic at each stage boundary |\n| Emotion / prosody | TTS adds prosody; quality varies by provider |\n| Cost | Pay for three separate model calls per turn |\n| Control | High: swap any component independently; use any LLM |\n| Open-weight path | Yes — each stage can run on open-weight models |\n\n## Architecture 2: Native speech-to-speech (realtime multimodal models)\n\nA single model ingests raw audio and outputs raw audio directly, without a text intermediate at the core inference step. Turn detection, interruption handling, and prosody are handled inside the model.\n\n**Tradeoffs:**\n\n| Dimension | Native speech-to-speech |\n|---|---|\n| Latency | Lower end-to-end (one model, streaming output) |\n| Interruptibility | Built into the model; lower barge-in latency |\n| Emotion / prosody | Richer; the model controls vocal tone end-to-end |\n| Cost | Single model call, but audio tokens are expensive |\n| Control | Lower: you cannot swap the underlying LLM independently |\n| Open-weight path | Limited — open-weight native speech-to-speech models are still emerging as of mid-2026 |\n\n## Vendor realtime APIs\n\nAll entries below are web-verified as of 2026-07-09.\n\n### OpenAI Realtime API\n\nA native speech-to-speech API. The original `gpt-realtime` (GA August 28, 2025) was superseded by **`gpt-realtime-2`** (May 7, 2026 — GPT-5-class reasoning, configurable reasoning effort, 128K context), then by **`gpt-realtime-2.1`** and mini variant **`gpt-realtime-2.1-mini`** (July 6, 2026 — improved alphanumeric recognition, noise/silence handling, and interruption behavior; ~25% lower p95 latency via caching). List pricing for 2.1 is unchanged from gpt-realtime-2 / the original mini (audio ≈$32/$64 per 1M input/output tokens full model, ≈$10/$20 mini) — no price cut shipped alongside 2.1, despite some chatter to that effect. The earlier `gpt-4o-realtime-preview` series is deprecated.\n\nTransports: **WebRTC** (recommended for browsers and mobile — lower jitter, handles NAT traversal) and **WebSocket** (recommended for server-to-server). A **SIP** integration path is also available for telephony.\n\nSupports streaming audio input and output, tool/function calling mid-conversation, VAD and server-side turn detection, and barge-in. Approximate glass-to-glass latency: 300–600 ms on subsequent turns.\n\nDocs: platform.openai.com/docs/guides/realtime-webrtc, developers.openai.com/api/docs/guides/realtime-websocket, and developers.openai.com/api/docs/models/gpt-realtime-2.1\n\n### Google Gemini Live API\n\nA native speech-to-speech API with bidirectional streaming over **WebSocket**. The model processes audio input and returns audio output natively, without a text intermediate. GA model: Gemini 2.5 Flash (native audio), available via both Google AI for Developers and Vertex AI. A newer **Gemini 3.1 Flash Live** (preview, released March 26, 2026) adds sharper acoustic-nuance detection and lower latency, but as of this writing is available only via Google AI Studio — no Vertex AI availability or GA date has been announced.\n\nSupports multimodal input (audio + video/screen), turn detection, barge-in, and function calling.\n\nDocs: ai.google.dev/gemini-api/docs/live-api\n\n### Amazon Nova Sonic\n\nA native speech-to-speech model on Amazon Bedrock, announced April 2025. The current generation is **Amazon Nova 2 Sonic** (December 2025). Accessed via Bedrock's **bidirectional streaming API** (WebSocket). Also supports WebRTC via an AWS blog reference implementation.\n\nSupports tool use, voice selection, interruption handling, and background-noise robustness. Integrates with Amazon Connect and telephony providers (Vonage, Twilio) and open frameworks including LiveKit and Pipecat.\n\nDocs: docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html\n\n### xAI Grok Voice Agent API\n\nA realtime speech-to-speech API launched December 17, 2025. Uses **bidirectional WebSocket** streaming. Compatible with the OpenAI Realtime API specification, so clients built for OpenAI Realtime can point at xAI with minimal changes.\n\nFeatures: custom VAD, Smart Turn end-of-turn detection, sub-1-second time-to-first-audio, 100+ language support with automatic detection. Also available via a native LiveKit plugin.\n\nDocs: docs.x.ai/docs/guides/voice\n\n## Open frameworks and orchestrators\n\n### Pipecat (pipecat-ai)\n\nOpen-source Python framework (BSD-2-Clause) for building real-time voice and multimodal conversational agents, developed by Daily. Organizes processing as pipeline frames flowing through transport, STT, LLM, and TTS stages. Supports 20+ STT providers and 30+ TTS providers, plus direct integrations with native speech-to-speech services (OpenAI Realtime, Amazon Nova Sonic, Gemini Live).\n\nTransports: WebRTC (Daily, LiveKit, SmallWebRTC), WebSocket, telephony. Handles VAD, turn detection, barge-in, and multi-agent coordination.\n\nGitHub: github.com/pipecat-ai/pipecat\n\n### LiveKit Agents\n\nOpen-source Python and TypeScript framework (Apache 2.0) for building realtime voice, video, and physical AI agents on top of the LiveKit WebRTC infrastructure. SDK v1.0 GA April 2025 (current line: v1.6.x as of mid-2026).\n\nThe 1.0 release replaced the older **VoicePipelineAgent** and **MultimodalAgent** classes (both now deprecated) with a single unified orchestrator, **AgentSession**, which covers cascaded (STT → LLM → TTS) and native speech-to-speech backends (e.g. OpenAI Realtime, Gemini Live) without changing application code when switching between them. Includes built-in turn detection, barge-in, native MCP tool support, and function calling. Bring-your-own STT, LLM, and TTS with no lock-in.\n\nDocs: docs.livekit.io/agents\n\n## STT and TTS component vendors\n\n**STT:**\n\n- **Deepgram** — streaming STT via WebSocket. Nova-3 is the flagship model (low WER). Flux is a conversational STT model with model-integrated end-of-turn detection and configurable turn-taking dynamics, designed specifically for voice agent pipelines. Docs: developers.deepgram.com\n\n- **whisper.cpp** (ggml-org) — C/C++ port of OpenAI's Whisper ASR models; runs locally with no external dependencies. Supports GPU acceleration and VAD. Use for on-device or self-hosted STT when latency from network round-trips to a cloud STT API is a constraint. GitHub: github.com/ggml-org/whisper.cpp\n\n**TTS:**\n\n- **ElevenLabs** — streaming TTS via WebSocket. Eleven v3 model. Broadest language coverage (70+ languages), strong voice cloning. Streaming endpoint: `/v1/text-to-speech/{voice_id}/stream-input`. Docs: elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-input\n\n- **Cartesia** — streaming TTS via WebSocket. **Sonic-3.5** (released June 2026) is the current flagship — approx. 40 ms time-to-first-audio on the Turbo variant, sub-100 ms on standard, native support for 42 languages. The prior Sonic 3 model remains available. Optimized for streaming latency at scale. Docs: docs.cartesia.ai\n\n## Key concepts\n\n- **VAD (Voice Activity Detection)** — a classifier that scores incoming audio for voice presence, typically using an energy threshold plus a voice classifier plus a minimum-duration guard. Determines when the user is speaking.\n- **End-of-turn detection / endpointing** — deciding that the user has finished their utterance and the agent should respond. Can be energy-based (silence duration), STT-integrated (model signals completion), or model-based (an ML classifier predicts semantic completion rather than just silence). Model-based detection has lower latency than waiting for silence timeouts.\n- **Barge-in / interruption handling** — detecting when the user speaks while the agent is responding, flushing the TTS output buffer, and restarting the listen/respond cycle. Failure to handle barge-in correctly is the most common CSAT failure in voice agents — distinguish genuine interruptions from backchannels (\"uh-huh\", \"right\") to avoid cutting off mid-sentence unnecessarily.\n- **Latency budget** — target sub-600 ms glass-to-glass for natural-feeling turn-taking. In a cascaded pipeline, the budget is roughly: STT (50–150 ms) + LLM time-to-first-token (100–300 ms) + TTS time-to-first-audio (50–150 ms). Each component must be streaming — do not wait for full STT transcript before starting LLM inference. See /resources/agent-cost-latency-optimization.\n- **Tool calling in a voice loop** — voice agents can call tools mid-conversation, but tool latency adds directly to voice latency. Keep tool calls under 200 ms; use speculative execution for predictable tool calls; return partial results to the model via streaming where possible. See /resources/reliable-tool-calling.\n\n## Practical guidance\n\n- **Measure end-to-end latency** per component (STT → LLM → TTS), not just total. Instrument each stage separately; bottlenecks are rarely where intuition points.\n- **Handle interruptions before tuning latency** — a fast agent that cannot be interrupted is worse than a slightly slower one that can.\n- **Keep tool calls fast** — tool latency is voice latency. Parallelize independent tool calls; cache results of stable lookups.\n- **Plan for transcription errors** — cascade pipelines inherit STT errors. Design prompts and tool schemas to tolerate common transcription noise (homophones, dropped words).\n- **Budget for always-on audio cost** — native speech-to-speech APIs price audio tokens at rates 3–10× higher than text tokens. Profile real traffic before committing to a pricing model.\n- For latency optimization across the full agent stack, see /resources/agent-cost-latency-optimization.\n- For tool-calling reliability in a voice loop, see /resources/reliable-tool-calling.\n- For framework choices when adding voice to a multi-agent system, see /resources/agent-frameworks-compared.\n\n## Verified sources\n\n- OpenAI Realtime API (WebRTC): https://platform.openai.com/docs/guides/realtime-webrtc\n- OpenAI Realtime API (WebSocket): https://developers.openai.com/api/docs/guides/realtime-websocket\n- OpenAI gpt-realtime GA announcement: https://openai.com/index/introducing-gpt-realtime/\n- OpenAI gpt-realtime-2.1 / gpt-realtime-2.1-mini announcement (July 6, 2026): https://community.openai.com/t/new-realtime-models-on-the-api-gpt-realtime-2-1-and-gpt-realtime-2-1-mini/1385896\n- gpt-realtime-2.1 pricing unchanged from gpt-realtime-2/original mini (no cut): https://www.marktechpost.com/2026/07/06/openai-gpt-realtime-2-1-mini-reasoning-realtime-api/\n- Google Gemini Live API overview: https://ai.google.dev/gemini-api/docs/live-api\n- Google Gemini 3.1 Flash Live (preview, March 26, 2026): https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview\n- Google Gemini Live API (WebSocket get started): https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket\n- Amazon Nova Sonic announcement (April 2025): https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-sonic-speech-to-speech-conversations-bedrock/\n- Amazon Nova 2 Sonic announcement (December 2025): https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-nova-2-sonic-real-time-conversational-ai\n- Amazon Nova Sonic bidirectional streaming docs: https://docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html\n- xAI Grok Voice Agent API announcement: https://x.ai/news/grok-voice-agent-api\n- xAI Voice Agent API docs: https://docs.x.ai/docs/guides/voice\n- Pipecat (pipecat-ai, BSD-2-Clause): https://github.com/pipecat-ai/pipecat\n- LiveKit Agents docs: https://docs.livekit.io/agents/voice-agent/\n- LiveKit Agents source (v1.6.x, Apache-2.0, confirms AgentSession as the current API): https://github.com/livekit/agents\n- Deepgram Nova-3 and Flux STT docs: https://developers.deepgram.com/docs/models-languages-overview\n- whisper.cpp (ggml-org, C/C++ Whisper port): https://github.com/ggml-org/whisper.cpp\n- ElevenLabs TTS WebSocket API: https://elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-input\n- Cartesia TTS WebSocket API: https://docs.cartesia.ai/api-reference/tts/websocket\n- Cartesia Sonic-3.5 / Ink-2 launch (June 2026): https://cartesia.ai/launch",
  "sources": [
    "https://platform.openai.com/docs/guides/realtime-webrtc",
    "https://developers.openai.com/api/docs/guides/realtime-websocket",
    "https://openai.com/index/introducing-gpt-realtime/",
    "https://community.openai.com/t/new-realtime-models-on-the-api-gpt-realtime-2-1-and-gpt-realtime-2-1-mini/1385896",
    "https://www.marktechpost.com/2026/07/06/openai-gpt-realtime-2-1-mini-reasoning-realtime-api/",
    "https://ai.google.dev/gemini-api/docs/live-api",
    "https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview",
    "https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket",
    "https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-sonic-speech-to-speech-conversations-bedrock/",
    "https://aws.amazon.com/about-aws/whats-new/2025/12/amazon-nova-2-sonic-real-time-conversational-ai",
    "https://docs.aws.amazon.com/nova/latest/userguide/speech-bidirection.html",
    "https://x.ai/news/grok-voice-agent-api",
    "https://docs.x.ai/docs/guides/voice",
    "https://github.com/pipecat-ai/pipecat",
    "https://docs.livekit.io/agents/voice-agent/",
    "https://github.com/livekit/agents",
    "https://developers.deepgram.com/docs/models-languages-overview",
    "https://github.com/ggml-org/whisper.cpp",
    "https://elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-stream-input",
    "https://docs.cartesia.ai/api-reference/tts/websocket",
    "https://cartesia.ai/launch"
  ]
}