# Voice AI Agent in Production: Architecture, Latency, Handoff and Launch

> How to build and run a voice AI agent in production: cascaded vs speech-to-speech architecture, turn detection and barge-in, the latency budget, tool calls in a voice loop, transport, cost, human handoff and launch sequencing.

Guide: Voice agents in production (pillar)
Published: 2026-10-07 · Updated: 2026-10-25 · 5456 words · ~7256 tokens (estimate)
Canonical: https://changegamer.ai/articles/voice-ai-agent-in-production
JSON: https://changegamer.ai/api/articles/voice-ai-agent-in-production.json

## In short

- A voice AI agent is a software agent that hears a caller, decides what to do, and answers aloud within a conversational turn, which makes turn-taking and latency the engineering problems that text agents never face.
- Production voice AI agents follow one of two architectures: a cascaded speech-to-text, language-model and text-to-speech pipeline that maximizes control, or a native speech-to-speech model that lowers latency at the price of vendor lock-in and expensive audio tokens.
- The corpus latency target for a natural-feeling voice agent is under 600 ms glass-to-glass, and every tool call made mid-conversation spends part of that budget, so tool latency is voice latency.
- A voice AI agent should be able to be interrupted correctly before anyone tunes it for speed, because an agent that cannot be cut off is worse than a slightly slower one that can.
- Live voice agents and offline dubbing are different engineering problems: dubbing works on a finished file with no barge-in or streaming deadline, so this guide covers only live caller conversations.
- As of October 2026 this site has no first-hand voice deployment: ChangeGamer does not run a voice agent, and the guidance here is synthesized from its reference corpus rather than from its own call traffic.

---

A voice AI agent has to be fast, interruptible and honest about what it does not know, all inside the same second, and most of the engineering effort goes into the first two. This guide is the hub for a cluster on running an AI voice agent in production: it covers the architecture decision, turn-taking, the latency budget, tools, transport, cost, handoff and launch order, and it points to reference pages for the contracts underneath.

Its scope is deliberately narrow. It covers live conversations with a caller, not offline dubbing or subtitling, which is a batch problem owned by [AI dubbing, localization and subtitling](/resources/ai-dubbing-localization-and-subtitling). It does not re-derive how to score a voice agent, which [evaluating voice agents](/resources/evaluating-voice-agents) owns. And it does not claim telephony expertise; the corpus this site draws on says very little about phone networks, and the section on transport says exactly how little.

## What is a voice AI agent, and how do you build one for production?

A voice AI agent is an agent that listens to a person speaking, decides what to say or do, and replies aloud fast enough to feel like conversation, and building one for production means choosing an architecture, getting turn-taking right, holding a latency budget, and wiring in tools, handoff and monitoring before launch. The site's [voice and realtime agents](/resources/voice-realtime-agents) reference, updated 2026-09-28, calls real-time voice one of the fastest-growing agent deployment patterns of 2026 and names two dominant architectures, which is the first decision below.

The build order that follows from the corpus, with the reasoning behind each step marked where it is this guide's own inference:

1. **Pick the architecture** (cascaded or native speech-to-speech), since vendor and framework options all depend on it.
2. **Get interruption handling working** before tuning speed, since the reference ranks an agent nobody can interrupt below a somewhat slower one that yields the floor.
3. **Set a latency budget and instrument each stage separately**, because bottlenecks are rarely where intuition points.
4. **Constrain tools** so a lookup cannot blow the budget.
5. **Design the human handoff** before the first real caller reaches the agent.
6. **Evaluate offline, gate the release, then monitor online** for what the offline set missed.

Two things make voice harder than a chat agent, and both are structural rather than a matter of polish. First, there is no reading time: a text user can wait three seconds without noticing, whereas a caller experiences three seconds of silence as a broken line. Second, the input is not text. A mis-heard word upstream corrupts everything after it, as the evaluating-voice-agents reference notes when it lists a mis-transcribed request among the failures a text-only evaluation never surfaces.

The rest of this guide takes those six steps in order, with a table or checklist wherever a decision has more than two options.

## Cascaded pipeline or native speech-to-speech: which architecture should you choose?

Choose a cascaded pipeline if you need control over each component, and a native speech-to-speech model if low latency and built-in interruption handling matter more than portability. The voice-realtime-agents reference describes both, and the trade-offs in the table below are its trade-offs restated as a decision aid rather than a new analysis.

| Question | Cascaded (STT, LLM, TTS) | Native speech-to-speech |
|---|---|---|
| Stages per turn | Three models in sequence | One model |
| Latency | Higher; sub-second needs fast transcription, a cached prompt prefix and streaming synthesis | Lower end to end |
| Interruption handling | You build it at each stage boundary | Inside the model |
| Cost shape | Three model calls per turn | One call, but audio tokens are expensive |
| Swap the language model | Yes, independently | No |
| Open-weight route | Yes, per stage | Limited as of mid-2026 |

### When the cascaded pipeline wins

The cascaded pipeline wins when your constraints are about control and data rather than raw speed. Each stage is a seam where you can log, redact, validate or replace something, and the stages can run on open-weight models if policy requires. The cost is that you own turn detection and barge-in, and you pay for three sequential model calls on every turn.

### When native speech-to-speech wins

Native speech-to-speech wins when natural-feeling timing is the product and you can accept one vendor's model underneath it. The reference credits it with richer prosody, because the model controls tone end to end, and with interruption behavior that is built in rather than assembled. Its stated costs are lock-in on the underlying model and audio tokens priced well above text; the reference puts native audio pricing at roughly 3 to 10 times text-token rates and advises profiling real traffic before committing to a pricing model.

### A practical way to decide

This is inference rather than a corpus claim, but the decision usually reduces to two questions. Do you need to inspect or alter the text between hearing and speaking, for example to redact account numbers or run an output filter? If so, the cascaded seams are valuable. Is your conversation short, transactional and timing-sensitive, with little need to touch the intermediate text? Then the single-model route deserves a trial. Because open frameworks such as LiveKit Agents document one orchestrator covering both backends without application changes, a prototype on one architecture does not have to be a permanent commitment, which lowers the cost of getting this decision wrong.

## How do turn detection and barge-in work, and why do they come first?

Turn detection decides when the caller has finished speaking, and barge-in handling decides what happens when the caller speaks over the agent, and both come before latency tuning because the reference ranks an uninterruptible fast agent below a somewhat slower interruptible one. In a cascaded pipeline these are explicit engineering work; in a native speech-to-speech system they are handled inside the model.

### The pieces

The corpus names three related mechanisms:

- **Voice activity detection (VAD)** classifies incoming audio for the presence of speech, typically combining an energy threshold, a voice classifier and a minimum-duration guard.
- **End-of-turn detection**, also called endpointing, decides the caller is done and the agent should respond. It can be energy-based (waiting out a silence), integrated into the speech-to-text stage, or model-based, where a classifier predicts semantic completion rather than just silence. The reference states that model-based detection has lower latency than waiting for silence timeouts.
- **Barge-in** detects the caller speaking while the agent talks, flushes the pending synthesized audio and restarts the listen-and-respond cycle.

### Backchannels are the hard part

Barge-in is not simply "stop talking when you hear sound." The reference warns that failure to handle it correctly is the most common satisfaction failure in voice agents, and that genuine interruptions have to be told apart from backchannels such as "uh-huh" or "right", so the agent does not cut itself off mid-sentence for no reason. The evaluating-voice-agents reference turns this into two measurable quantities, barge-in accuracy and false-interruption rate, which is why the launch checklist later in this guide includes both.

### What this means for the build

If you use a native model, test its interruption behavior with realistic overlapping speech before anything else, since you cannot tune what lives inside the model. If you build a cascade, treat turn detection as a component with its own acceptance test, not a default setting. As of July 2026, the corpus lists Deepgram's Flux, a transcription model that decides end of turn itself, and xAI's voice API as offering its own Smart Turn detection; both are examples of vendors absorbing this problem so that you do not have to build it.

## What latency budget should a voice agent hold?

A voice agent should hold under 600 ms glass to glass, which is the target the voice-realtime-agents reference gives for natural-feeling turn-taking. That is the elapsed time from the end of the caller's speech to the start of the agent's audio reply as the caller experiences it.

### How the budget divides in a cascade

The reference gives a rough split for a cascaded pipeline, shown here with the sum worked out:

| Stage | Reference range |
|---|---|
| Speech-to-text | 50 to 150 ms |
| Language model, time to first token | 100 to 300 ms |
| Speech synthesis, time to first audio | 50 to 150 ms |
| **Sum of the three** | **200 to 600 ms** |

The sum is this guide's arithmetic, and it carries a warning that the reference does not spell out in so many words: at the top of every range the stages consume the entire 600 ms, leaving nothing for network transit, turn-detection delay or a tool call. Treat the table as a statement that the budget is tight, not that it is comfortably met.

### Rules that follow

- **Stream every stage.** The reference says not to wait for the full transcript before starting model inference, and each component must stream.
- **Instrument per stage, not just in total.** Measure speech-to-text, model and synthesis separately, and report percentiles, since the evaluation reference suggests P50 and P95 for glass-to-glass latency.
- **Reserve headroom for tools.** Anything the agent calls mid-turn subtracts directly from the same 600 ms.
- **Do not trust vendor numbers as your numbers.** As of July 2026 the corpus reports roughly 300 to 600 ms glass to glass on subsequent turns for OpenAI's realtime API and sub-1-second time-to-first-audio for xAI's, but those are vendor-side or approximate figures and your caller's network, region and prompt size will move them.

For the cost and latency levers that apply across the whole agent stack rather than the voice loop alone, the site's [agent cost and latency optimization](/resources/agent-cost-latency-optimization) reference is the place to look. For what a deadline should do when a stage overruns, see [timeouts and deadlines for AI agents](/articles/timeouts-and-deadlines-for-ai-agents).

## How should a voice agent call tools without breaking the latency budget?

A voice agent should keep each mid-conversation tool call fast and predictable, because the voice-realtime-agents reference states that tool latency adds directly to voice latency and recommends keeping calls under 200 ms. Vendor realtime APIs from OpenAI, Google, Amazon and xAI all support function calling alongside streaming audio as of the reference's July 2026 verification, so the capability is not the constraint; the clock is.

### Tactics the reference supports

- Use speculative execution for predictable calls, starting a likely lookup before the model asks for it.
- Parallelize independent calls and cache the results of stable lookups.
- Stream partial results back to the model where the tool allows it.

The general mechanics of schema validation, retries and idempotency live in the [reliable tool calling](/resources/reliable-tool-calling) reference and in the articles on [retries and idempotency for AI agents](/articles/retries-and-idempotency-for-ai-agents) and [tool-calling contracts for AI agents](/articles/tool-calling-contracts-for-ai-agents). A voice-specific caution, offered as inference: a retry that would be invisible in a chat window is an audible pause on a call, so retry budgets in voice should be tighter than in text, and the agent needs something sensible to say while it waits.

### What to say while a tool runs

This is inference, not a corpus claim, but silence is the one thing a call cannot afford. A short spoken acknowledgment ("let me check that") fills the gap for a slow lookup, while a lookup you expect to finish inside the budget should simply run without commentary, because filler on every turn becomes its own annoyance. Decide per tool which category it falls in, and write that decision down next to the tool's schema.

### Transcription noise reaches your tools

Cascaded pipelines inherit speech-to-text errors, and the reference advises designing prompts and tool schemas to tolerate common noise such as homophones and dropped words. In practice that means tool arguments that came from speech, such as names, numbers and identifiers, deserve confirmation before any action that is hard to undo. That confirmation step costs a turn, so spend it only where a wrong value is expensive.

## Which transport should a voice agent use: WebRTC, WebSocket or SIP?

Transport should follow where the caller is: WebRTC for browser and mobile clients, WebSocket for server-to-server integration, and a SIP path where a vendor offers one for telephony. That is the pattern in the voice-realtime-agents reference, which words it as a tendency rather than a rule.

| Transport | Typical use per the corpus | Notes from the corpus |
|---|---|---|
| WebRTC | Browser and mobile clients | OpenAI recommends it for lower jitter and NAT traversal handling |
| WebSocket | Server-to-server | Google Gemini Live, Amazon Nova Sonic and xAI use bidirectional WebSocket streaming |
| SIP | Telephony | OpenAI is documented as offering a SIP path; no detail beyond that |

For the per-vendor transport matrix and a decision tree by client type, see [WebRTC vs WebSocket for voice agents](/articles/voice-agent-transport-webrtc-vs-websocket-vs-sip).

### What the corpus does not tell you about telephony

Telephony depth is not covered here, and that gap is real. The corpus says that some vendors add a SIP path and that Amazon Nova Sonic integrates with Amazon Connect and with Vonage and Twilio. It says nothing about number provisioning, trunking, how touch-tone input is handled, which audio codecs survive a phone call, or how the public phone network affects latency and audio quality, and this guide makes no claims on any of them. If your agent answers phone calls, treat the vendor's telephony documentation as the authority and plan a dedicated test with real calls, because the latency numbers above were not reported for phone paths.

### Frameworks and transport

The open frameworks absorb much of this choice. Pipecat lists WebRTC (Daily, LiveKit, SmallWebRTC), WebSocket and telephony transports, and LiveKit Agents builds on LiveKit's WebRTC infrastructure. Picking a framework therefore partly picks a transport menu, which is one more reason to decide the architecture first and the framework second.

## Should you build on a vendor realtime API, an open framework, or both?

Use a vendor realtime API when you want the shortest route to a working native speech-to-speech agent, an open framework when you want to own orchestration and keep components swappable, and both when the framework sits in front of the vendor model. The corpus describes each, and the combination is common because the frameworks list the vendor models as supported backends.

### Vendor realtime APIs as of July 2026

The voice-realtime-agents reference marks these entries as web-verified on 2026-07-09 and states that its 2026-09-28 pass could not re-verify the vendor pages, so model names and prices below carry that caveat.

| Vendor | What the corpus records |
|---|---|
| OpenAI Realtime | Native speech-to-speech; gpt-realtime reached general availability on 28 August 2025, then gpt-realtime-2 (7 May 2026) and gpt-realtime-2.1 with a mini variant (6 July 2026); WebRTC, WebSocket and SIP; roughly 300 to 600 ms glass to glass on later turns |
| Google Gemini Live | Native speech-to-speech over WebSocket; Gemini 2.5 Flash native audio is the generally available model, with Gemini 3.1 Flash Live in preview since 26 March 2026 via Google AI Studio only |
| Amazon Nova Sonic | Native speech-to-speech on Bedrock over bidirectional streaming; Nova 2 Sonic announced December 2025; works with Amazon Connect, Vonage, Twilio, LiveKit and Pipecat |
| xAI Grok Voice Agent API | Launched 17 December 2025; WebSocket; compatible with the OpenAI Realtime specification; sub-1-second time-to-first-audio; 100+ languages |

The OpenAI price detail in the reference is worth knowing before you budget: list audio pricing is about 32 and 64 dollars per million input and output tokens for the full model and about 10 and 20 for the mini, and the reference notes that no price cut shipped with version 2.1 despite chatter to that effect.

### Open frameworks

Pipecat is a BSD-2-Clause Python framework from Daily that models processing as frames moving through transport, speech-to-text, language-model and synthesis stages, with support for 20+ transcription and 30+ synthesis providers. LiveKit Agents is an Apache 2.0 Python and TypeScript framework whose AgentSession orchestrator covers both cascaded and native backends without application code changes when you switch; the reference lists version 1.8.3 dated 23 September 2026. Both bring their own turn detection and barge-in handling.

### How to decide, as inference

If you would regret being tied to one model vendor in two years, start with a framework, since the reference describes LiveKit Agents as bring-your-own with no lock-in. If you are validating whether a voice channel is worth building at all, a single vendor API is the cheapest experiment. For broader framework trade-offs beyond voice, see [agent frameworks compared](/resources/agent-frameworks-compared).

## Which speech-to-text and text-to-speech components matter in a cascaded stack?

In a cascaded stack the components that matter are a streaming transcription model with good end-of-turn behavior and a streaming synthesis model with low time to first audio, and the corpus lists specific options for each as of July 2026. Everything below is vendor-reported or reference-reported rather than benchmarked by this site.

### Speech-to-text

- **Deepgram** offers streaming transcription over WebSocket; Nova-3 is its flagship model, and Flux is a conversational model with end-of-turn detection built in and configurable turn-taking dynamics, aimed at voice-agent pipelines.
- **whisper.cpp** is a C and C++ port of OpenAI's Whisper that runs locally with no external dependencies, supports GPU acceleration and VAD, and suits on-device or self-hosted transcription when a network round trip to a cloud service is the constraint.

### Text-to-speech

- **ElevenLabs** provides streaming synthesis over WebSocket with the Eleven v3 model, 70+ languages and voice cloning.
- **Cartesia** provides streaming synthesis over WebSocket; the reference records Sonic-3.5 as released in June 2026 with approximately 40 ms time to first audio on the Turbo variant, sub-100 ms on standard, and 42 languages.

### Selection criteria

Choose components against your own audio, not a leaderboard. The evaluation reference suggests scoring transcription with word error rate on a held-out set of your own recordings, and checking synthesis for intelligibility on the names, numbers and acronyms your business uses, since a base voice was not tuned on them. The same reference cautions that word error rate is a diagnostic rather than a pass-or-fail gate, because a transcript can carry residual errors and still convey everything the agent needs. Where the voice cloning offered by synthesis vendors is concerned, a decision about consent and disclosure belongs to the policy layer; the corpus does not cover it, and this guide does not either.

## What does a voice AI agent cost to run?

A voice AI agent costs three sequential model calls per turn in a cascade or one expensive audio-token call per turn in a native model, and the corpus gives the shape of the cost but no total-cost figure for a call. Anything quoted as a per-minute price for a whole agent in a blog post or sales deck is not something this site's corpus supports.

What the corpus does give is directional:

- A cascade pays for transcription, the language model and synthesis on every turn.
- Native speech-to-speech is one call, but audio tokens are priced at roughly 3 to 10 times text-token rates, per the reference, and the OpenAI list prices above show the scale for one vendor as of July 2026.
- Cached prompt prefixes help latency in a cascade, per the budget guidance, and the agent cost and latency reference covers caching economics generally.

### Measure per call, not per token

Inference follows. Because voice cost depends on conversation length, silence and how often callers interrupt, the unit worth tracking is cost per completed call, with its distribution rather than its average. The mechanics for recording spend per span, keeping failed runs in the books and reconciling against an invoice are in [agent cost telemetry in production](/articles/agent-cost-telemetry-in-production), and they apply to voice with one addition: audio and text tokens should be recorded as separate counts, since they are priced differently.

### Profile before you commit

The reference's advice is to profile real traffic before committing to a pricing model, and that is sound for a second reason: early test calls are shorter and cleaner than production calls. Run a pilot with real callers and real background noise before extrapolating a monthly figure.

## How should a voice agent hand a caller to a human?

A voice agent should hand a caller to a human on explicit, enumerable triggers, pass the full context along, and honor a request for a person immediately, which is the escalation design in the [AI customer support agents](/resources/customer-support-agents) reference applied to a live call. Voice raises the stakes of that design because a caller cannot skim or scroll; a bad handoff is audible and immediate.

### Escalation triggers

The customer-support reference lists trigger categories that carry over directly:

- **The caller asks for a person**, however phrased, and gets one without having to argue the point.
- **Sentiment or repeat-contact signals**, routing a frustrated or returning caller before the call sours.
- **Policy and account boundaries**, meaning money above a set limit, closing an account, legal or safety topics, and changes to payment or identity details.
- **Gaps in knowledge**, such as a lookup that found nothing, a tool that errored, or a question the agent was never scoped to answer.
- **Turn and retry caps**, where reaching the cap is itself the signal.

The reference also insists that escalation come from explicit signals rather than the model's own uncalibrated confidence. Voice adds one wrinkle that is this guide's inference: sentiment is partly carried by tone, and a cascaded pipeline discards tone once audio becomes text, so a team that depends on frustration signals should check whether its pipeline preserves them.

### Warm handoff, not cold

A warm handoff carries over the transcript, any fields already collected, half-finished tool state and why the call is escalating. A cold handoff drops the caller into a blank queue to start over, which the reference rates as worse than never having offered the agent. On a call, "please hold while I transfer you" followed by a human asking for the account number again is the failure to design against.

### Deflection is a definition, not a goal

The same reference cautions that a deflection count includes contacts closed without a human, so it cannot separate a real resolution from a caller who quit. It deliberately cites no vendor deflection or handle-time figures, and this guide follows suit. Pair any containment number with a resolution-quality signal. Log the trigger category for every handoff, since that log is the feedback loop for scope.

The call-level mechanics, meaning the transfer announcement, failure paths and a context payload, are in [voice agent human handoff](/articles/voice-agent-human-handoff).

## Which guardrails and security controls does a voice agent need?

A voice agent needs the same structural guardrails as any customer-facing agent, enforced independently of the prompt, plus attention to what it says aloud that cannot be retracted. The customer-support reference argues that tone and topic limits need both a system prompt and a separate output check, since trusting the model to hold its persona under pressure is a fragile control.

### What carries over

- **Do not let the agent bind the company** to refunds, service promises or legal statements beyond documented policy; send those to escalation.
- **Treat what the caller says as untrusted input.** Callers can try to talk an agent into commitments, and anything the agent reads aloud into its context from a lookup is the same injection surface as in text. The [agent guardrails](/resources/agent-guardrails) reference covers the general principle.
- **Defense in depth** for injection, credentials and least privilege belongs to the [agent security operations](/articles/agent-security-operations) cluster; a voice agent inherits all of it.

### What is different in voice, as inference

Output filters in text can block a message before the user sees it. In a streaming voice pipeline, audio may already be playing by the time a filter judges the sentence, so where the check runs relative to synthesis matters and has to be designed, not assumed. The corpus does not document how vendors handle output filtering in a streaming audio path, so verify this with whichever stack you choose before relying on it. Personal data is a related concern: transcripts and recordings contain account details, so the guidance in [data privacy and PII for AI agents](/articles/data-privacy-and-pii-for-ai-agents) and [redacting sensitive data from agent traces](/articles/redacting-sensitive-data-from-agent-traces) applies to anything you store from a call.

## How do you evaluate and monitor a voice agent before and after launch?

Evaluate a voice agent per pipeline stage and end to end before each release, then monitor the same signals online after it, and use the dedicated reference for the metrics themselves. This guide does not re-derive them, because [evaluating voice agents](/resources/evaluating-voice-agents) owns that contract.

### The shape of the plan

The evaluation reference separates offline from online work cleanly. Offline work replays a stored set of real or synthetic calls, deliberately including accents, noise and people talking over each other, ahead of each release; results repeat exactly but cannot reveal conditions absent from the set. Online work tracks transcription confidence, latency per stage and overall, and containment on live traffic, which surfaces drift the stored set never saw. The split it recommends: offline results decide whether to ship, online monitoring covers what remains.

### Metrics by stage, in brief

- Transcription: word error rate against a held-out set, and real-time factor for throughput.
- Orchestration: task completion without human handoff, and tool-call correctness.
- Synthesis: mean opinion score or an automated proxy, and intelligibility on domain terms.
- End to end: glass-to-glass percentiles against the 600 ms budget, barge-in accuracy and false-interruption rate.

The reference also names VoiceBench for general knowledge, instruction following and safety under accent, noise and speaking-rate variation, and notes that tau-cubed-bench added a full-duplex voice mode scored with pass^k. Those are starting points for building your own test set, not substitutes for it.

### Where the general agent guides apply

The release-gating discipline is in [how to evaluate AI agents in CI](/articles/evaluating-ai-agents-in-ci), and the live-traffic side, tracing, screening, online metrics and incident fixtures, is in [AI agent observability and the production evaluation playbook](/articles/agent-observability-and-evaluation). Voice adds audio-specific signals but not a new philosophy. One inference worth stating: a failed call is only reproducible if you kept the audio or at least the transcript and per-stage timings, so decide your retention and redaction policy before launch, not after the first incident.

## What goes wrong with voice agents in production?

The failures that matter most are the ones the corpus already names, plus a few that follow from combining it with general agent reliability, and each has a cheap detection signal. The table separates what the corpus states from what this guide infers.

| Failure | Basis | Early signal |
|---|---|---|
| Agent talks over the caller or is cut off by backchannels | Corpus: the most common satisfaction failure in voice | False-interruption rate, barge-in accuracy |
| Mis-heard request corrupts the downstream step | Corpus: inherited transcription errors | Word error rate on held-out calls; confirmation steps on critical values |
| Tool call stalls the turn | Corpus: tool latency is voice latency | Per-tool P95 latency within the turn |
| Latency creeps past the budget | Corpus: budget is tight when summed | End-to-end P50 and P95 per stage |
| Cold handoff forces the caller to repeat everything | Corpus: customer-support reference | Handoff logs missing structured fields |
| Deflection looks healthy while callers give up | Corpus: deflection is a definition | Repeat-contact rate alongside containment |
| Vendor model changes behavior silently | Inference, since the model names changed several times between 2025 and mid-2026 per the vendor table | Release replay on a fixed call set after any model change |
| Streaming audio plays before a guardrail judges it | Inference, not documented in the corpus | Test with adversarial scripts against the exact stack |

### Outages and degraded modes

If a vendor model, a transcription service or a helpdesk API fails mid-call, the agent needs a defined fallback, such as a clear spoken apology and a transfer to a human, rather than silence. The pattern is covered generally in [circuit breakers and degraded mode for AI agents](/articles/circuit-breakers-and-degraded-mode-for-ai-agents) and in [agent incident response runbooks](/articles/agent-incident-response-runbooks). The voice-specific addition, as inference, is that the degraded path has to be audible and fast: a call that goes quiet for ten seconds is abandoned.

### Abuse and load

Voice endpoints that answer anyone can be hit with automated or hostile calls, and each call costs real model spend. [Rate and abuse controls for AI agents](/articles/rate-and-abuse-controls-for-ai-agents) covers the principles; the corpus has no voice-specific abuse data, so any thresholds should come from your own traffic.

## How should you sequence a voice agent launch?

Sequence a voice agent launch by starting with a narrow set of low-risk request types, gating each release on offline replays, rolling out gradually, and expanding scope only when handoff and resolution logs say it is safe. The scoping advice comes from the customer-support reference, which recommends launching on well-understood request types such as order status, password reset and documented policy lookups before judgment-heavy categories.

A launch order consistent with the corpus and with the general rollout guidance:

1. **Prototype on one architecture** with a framework in front, so the choice stays reversible.
2. **Build the offline call set**, including noisy, accented and overlapping speech, and set pass criteria per stage.
3. **Instrument each stage** for latency percentiles and log every escalation with its trigger.
4. **Pilot with internal callers**, then a small share of real traffic, with a human fallback always reachable.
5. **Expand scope** only as deflection, repeat-contact and handoff quality all hold, never on deflection alone.
6. **Re-run the replay set on every vendor model change**, since model identifiers moved repeatedly in the period the reference covers.

Rollout mechanics, traffic splitting and rollback triggers are covered in [agent rollout and rollback](/articles/agent-rollout-and-rollback), and the live-traffic metrics and drift checks are in [online agent metrics and drift monitoring](/articles/online-agent-metrics-and-drift-monitoring). The general readiness checklist this guide does not repeat is in the [shipping agents to production](/resources/shipping-agents-to-production) reference.

## What does this guide not cover, and what is the evidence behind it?

Telephony depth, dubbing, voice-cloning policy and any voice deployment of ChangeGamer's own are outside the scope of this guide, whose evidence is the site's reference corpus rather than operating experience. Naming the limits is part of the point.

- **Telephony**: the corpus records only that some vendors offer a SIP path and lists telephony integrations for Amazon Nova Sonic; nothing on carriers, routing, touch-tone handling, codecs or the phone network.
- **Offline dubbing and subtitling**: a batch problem with its own pipeline, owned by [AI dubbing, localization and subtitling](/resources/ai-dubbing-localization-and-subtitling), which states that live-agent concerns like barge-in and streaming latency do not apply there.
- **Vendor re-verification**: the vendor facts above were verified by the corpus on 2026-07-09, and the 2026-09-28 pass confirmed only the open-framework versions, not the vendor APIs. Re-check model names, prices and latency claims before you commit.
- **No first-hand voice deployment**: ChangeGamer runs no voice agent and has no call traffic, so nothing here is a measured result from live callers.

### The one first-hand datapoint

The one thing ChangeGamer can report from its own measurements concerns demand rather than operation. In its 2026-10-01 ranking snapshot, the site's own reference page [voice and realtime agents](/resources/voice-realtime-agents) ranked 62 for the query "realtime voice ai", which the snapshot records at 140 searches a month. That is a position on one query on one date, not evidence about how well any voice agent performs, and it is the reason this cluster exists: the corpus already holds the contract, and the missing piece is the operator's decision guide.

### Where the cluster goes next

The cluster is complete as of 15 October 2026: all 12 sub-articles are published, and each owns a single intent and leans on the references named above rather than restating them. In cluster order:

1. [The voice agent latency budget](/articles/voice-agent-latency-budget)
2. [How to test turn detection and barge-in](/articles/voice-turn-detection-and-barge-in-testing)
3. [What to log and filter in a voice agent, by architecture](/articles/voice-agent-architecture-cascaded-vs-speech-to-speech)
4. [Function calling in a voice agent](/articles/tools-in-a-voice-agent-loop)
5. [Voice agent transport: WebRTC vs WebSocket, and where SIP fits](/articles/voice-agent-transport-webrtc-vs-websocket-vs-sip)
6. [Voice agent human handoff](/articles/voice-agent-human-handoff)
7. [How to compute your voice agent cost per minute](/articles/voice-agent-cost-per-minute)
8. [Voice agent guardrails for spoken output](/articles/guardrails-for-spoken-output-in-voice-agents)
9. [How to monitor a voice agent in production](/articles/voice-agent-online-monitoring)
10. [How to handle voice AI vendor model changes](/articles/voice-agent-vendor-change-management)
11. [The voice agent launch checklist](/articles/voice-agent-launch-checklist)
12. [How a voice agent handles callers who speak another language](/articles/voice-agent-multilingual-callers)

## Frequently asked questions

### What is the difference between a cascaded voice agent and a speech-to-speech voice agent?

A cascaded voice agent chains three separate models, speech-to-text, a language model and text-to-speech, so each stage can be swapped independently, while a speech-to-speech agent uses one model that takes in and produces audio directly, which lowers latency and handles interruptions internally but ties the team to that vendor and prices audio tokens at a premium. The choice is mainly control versus latency.

### How fast does an AI voice agent need to respond?

The reference target in this site's corpus is under 600 ms from the end of the caller's speech to the first audio of the reply, measured glass to glass, because slower responses stop feeling like natural turn-taking. In a cascaded pipeline that budget is shared by transcription, the first language-model token and the first synthesized audio, plus any tool call made during the turn.

### Can a voice AI agent answer phone calls?

Some vendors offer a SIP integration path for telephony, according to this site's voice reference as of July 2026, but this guide does not cover telephony depth such as carrier setup, call routing or touch-tone handling, so confirm phone-network details with the vendor you pick rather than relying on this page.

### Is a voice AI agent the same as AI dubbing or text-to-speech?

No. A voice AI agent holds a live two-way conversation under a sub-second latency budget, while AI dubbing re-voices a finished recording in a batch job with no turn-taking, and plain text-to-speech is only one stage inside a cascaded agent. Each has different failure modes, metrics and vendors, which is why they are separate topics here.

### How do you know when a voice AI agent should hand the call to a human?

Hand off on explicit, enumerable triggers rather than on the model's own sense of uncertainty: the caller asks for a person, the same issue recurs, the request crosses a policy boundary such as a large refund, retrieval or a tool call fails, or a turn or retry cap is reached. Log the trigger category with every handoff so it can be tuned later.


---

## Everything in this guide

- [Voice Agent Latency Budget: Where the 600 ms Goes and How to Hold It](https://changegamer.ai/articles/voice-agent-latency-budget.md): How to split a 600 ms voice agent latency budget across speech-to-text, the language model, speech synthesis and tool calls, how to measure each stage, and which levers hold the budget.
- [How to Test Voice Agent Turn Detection and Barge-In](https://changegamer.ai/articles/voice-turn-detection-and-barge-in-testing.md): How to test voice agent turn detection and barge-in: build a labelled clip set, count barge-in accuracy and false-interruption rate, separate cut-off from talk-over failures, and split offline gates from online monitors.
- [What to Log and Filter in a Voice Agent, by Architecture](https://changegamer.ai/articles/voice-agent-architecture-cascaded-vs-speech-to-speech.md): What to log and filter in a voice agent depending on architecture: where a cascaded pipeline gives you text seams for logging, redaction and output filters, what a native speech-to-speech model leaves you to do from outside, and what a model swap forces you to re-test.
- [Function Calling in a Voice Agent: What a Tool May Be in a Spoken Loop](https://changegamer.ai/articles/tools-in-a-voice-agent-loop.md): Function calling in a voice agent: which vendors and frameworks support tools as of the corpus's 2026-07-09 verification, how to classify tools as lookup, action or handoff, what barge-in does to a side effect, and a checklist for admitting a tool.
- [Voice Agent Transport: WebRTC vs WebSocket, and Where SIP Fits](https://changegamer.ai/articles/voice-agent-transport-webrtc-vs-websocket-vs-sip.md): WebRTC vs WebSocket for voice AI agents: which transport each vendor documents as of the corpus's 2026-07-09 verification, how cascaded stacks use WebSocket components, a decision tree by client type, and a per-transport test checklist.
- [Voice Agent Human Handoff: What Happens on the Call When the Agent Lets Go](https://changegamer.ai/articles/voice-agent-human-handoff.md): Voice agent human handoff mechanics: the handoff as a tool that ends the agent's session, barge-in during the transfer announcement, what the caller hears on hold and on failure, a context payload schema, and a real-call test checklist.
- [How to Compute Your Voice Agent Cost per Minute](https://changegamer.ai/articles/voice-agent-cost-per-minute.md): How to compute a voice agent cost per minute from your own pilot: symbolic formulas for native speech-to-speech and cascaded STT, LLM and TTS, the variables that move the result, idle audio, retries, tool-call turns, and a sensitivity check.
- [Voice Agent Guardrails: Checking What the Agent Says Before the Caller Hears It](https://changegamer.ai/articles/guardrails-for-spoken-output-in-voice-agents.md): Voice agent guardrails for spoken output: sentence-level gating before TTS, fail-closed judges, spoken numbers a text PII filter misses, readback before state changes, transcription-driven injection, refusal phrasing for the ear, and an adversarial test list.
- [How to Monitor a Voice Agent in Production: Dashboard, Alerts and Runbook](https://changegamer.ai/articles/voice-agent-online-monitoring.md): How to monitor a voice agent in production: per-stage latency percentiles, transcription confidence, containment paired with a resolution signal, live false-interruption rate, a handoff trigger review, and call-audio retention, each with what to chart, alert on and do.
- [How to Handle Voice AI Vendor Model Changes Without Regressing Callers](https://changegamer.ai/articles/voice-agent-vendor-change-management.md): How to handle voice AI vendor model changes: classify the change, record the model id on every trace, replay an offline set with hard calls on old and new, canary before cutover, keep a rollback, and run a deprecation-readiness routine.
- [Voice Agent Launch Checklist: The Go/No-Go Gates Before Real Callers](https://changegamer.ai/articles/voice-agent-launch-checklist.md): A voice agent launch checklist as yes/no gates: scope, offline replay, per-stage latency, barge-in, tools, transport, handoff, cost, guardrails, monitoring and rollback, each with a pass signal and the article that owns the method.
- [How Does a Voice Agent Handle Callers Who Speak Another Language?](https://changegamer.ai/articles/voice-agent-multilingual-callers.md): How a live voice agent handles callers who speak another language: detecting the language, choosing speech components per language, slicing evaluation by language and handing off when the language is unsupported, with the thin corpus evidence stated plainly.

## Reference resources

- https://changegamer.ai/resources/voice-realtime-agents.md
- https://changegamer.ai/resources/evaluating-voice-agents.md
- https://changegamer.ai/resources/ai-dubbing-localization-and-subtitling.md
- https://changegamer.ai/resources/customer-support-agents.md
- https://changegamer.ai/resources/agent-cost-latency-optimization.md
- https://changegamer.ai/resources/reliable-tool-calling.md
- https://changegamer.ai/resources/agent-guardrails.md
- https://changegamer.ai/resources/evaluating-ai-agents.md
- https://changegamer.ai/resources/agent-frameworks-compared.md
- https://changegamer.ai/resources/shipping-agents-to-production.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt
Licensing: https://changegamer.ai/api/pricing.json (offer catalog) · https://changegamer.ai/api/payment.json (payment methods, HTTP 402 flow) · access guide: https://changegamer.ai/resources/access-and-pricing.md