Voice Agent Latency Budget: Where the 600 ms Goes and How to Hold It
How to split a 600 ms voice agent latency budget across speech-to-text, the language model, speech synthesis and tool calls, how to measure each stage, and which levers hold the budget.
- A cascaded voice agent splits its sub-600 ms target across speech-to-text at 50 to 150 ms, language-model first token at 100 to 300 ms and speech synthesis first audio at 50 to 150 ms, per the voice-realtime-agents reference.
- Adding the three reference ranges gives 200 ms at the bottom and 600 ms at the top, so a voice agent running at the top of every range has already spent the entire budget before any tool call or network hop.
- The 600 ms figure is a target for natural-feeling turn-taking, not a measurement, so a team should record P50 and P95 for each pipeline stage and for the whole turn rather than assume the target is met.
- A mid-turn tool call that takes the 200 ms ceiling the reference recommends consumes a third of the 600 ms budget, which is why tool latency is treated as voice latency.
- Vendor latency figures such as OpenAI at roughly 300 to 600 ms on later turns are vendor-reported, were verified on 2026-07-09 and were not re-verified on 2026-09-28, so they should not substitute for measurements on your own traffic.
A voice agent latency budget is a 600 ms allowance, from the end of the caller's speech to the first audio of the reply, that has to be divided among transcription, the language model, speech synthesis and any tool call. This article covers how to split that allowance, how to measure it and which levers hold it. It is about live caller conversations only: offline dubbing has no turn-taking deadline, and how to score a voice agent lives in evaluating voice agents. For the architecture choice, turn detection and handoff, start from the pillar guide.
How is a 600 ms voice latency budget split across stages?
A cascaded pipeline divides the budget among three stages, and the voice-realtime-agents reference (updated 2026-09-28) gives a range for each. The sums in the last row are the author's arithmetic, not a corpus figure.
| Stage | Range in the reference | Best case | Worst case |
|---|---|---|---|
| Speech-to-text | 50 to 150 ms | 50 ms | 150 ms |
| Language model, first token | 100 to 300 ms | 100 ms | 300 ms |
| Speech synthesis, first audio | 50 to 150 ms | 50 ms | 150 ms |
| Sum (author's arithmetic) | 200 ms | 600 ms |
Two cautions follow from the table. First, the sub-600 ms figure is a target for natural-feeling turn-taking, not something anyone has measured on your system. Second, the worst case equals the target exactly, so the ranges leave no room for anything outside the three models. The reference gives no figure for network transit or the delay before the system decides the caller has stopped talking, and this article does not invent one.
What happens to the budget when a tool call is added?
A tool call in the middle of a turn adds its full duration to the caller's wait, because the agent cannot speak the answer until the result is back. The reference says tool latency adds directly to voice latency and recommends keeping calls under 200 ms. The scenarios below are the author's arithmetic, assuming the tool runs in series with the three stages and takes the 200 ms ceiling.
| Scenario (author's arithmetic) | STT | First token | First audio | Tool | Total |
|---|---|---|---|---|---|
| Baseline cascade, bottom of every range | 50 | 100 | 50 | 0 | 200 ms |
| Baseline cascade, midpoint of every range | 100 | 200 | 100 | 0 | 400 ms |
| Baseline cascade, top of every range | 150 | 300 | 150 | 0 | 600 ms |
| Plus one 200 ms tool call, bottom of ranges | 50 | 100 | 50 | 200 | 400 ms |
| Plus one 200 ms tool call, midpoint | 100 | 200 | 100 | 200 | 600 ms |
| Plus one 200 ms tool call, top of ranges | 150 | 300 | 150 | 200 | 800 ms |
The top of the range spends the whole 600 ms with no tool call at all. Even a mid-range cascade reaches the target exactly once a single compliant tool call joins the turn. As inference, a team that needs tools on most turns should plan around the lower half of every stage range, or accept filler speech while the tool runs (the pillar covers that decision).
Which levers hold the budget?
The levers that hold the budget are streaming at every stage, a cached prompt prefix, fast and parallel tools, and, where the architecture allows it, moving work off the network path. Each comes from the corpus and none has a measured gain attached here.
Stream every stage and cache the prefix
Each component must stream, and the language model should not wait for the full transcript before it starts, per the voice-realtime-agents reference. The same reference says that sub-second operation in a cascade needs fast transcription, a cached language-model prefix and streaming synthesis. Caching a stable prompt prefix is, per agent cost and latency optimization, typically the highest-leverage token-side fix when the same instructions repeat on every call, and a voice agent repeats its system prompt on every turn. That reference notes that the discount and minimum prompt size differ per vendor.
Keep tools short and concurrent
The reference's tool tactics are speculative execution for predictable calls, parallel independent calls, cached results for stable lookups and partial results streamed back to the model. In latency terms, the agent cost and latency reference says running independent calls concurrently turns a sum into a maximum. Retries, schemas and idempotency belong to reliable tool calling and are not repeated here. For what should happen when a tool overruns, see timeouts and deadlines for AI agents.
Move transcription closer, or change architecture
When a cloud round trip is the bottleneck, the reference points to whisper.cpp for local transcription. It gives no latency figure for it, so measure before relying on it. Native speech-to-speech replaces the three stages with one model and is described as lower in end-to-end latency, but it brings vendor lock-in and audio tokens priced at 3 to 10 times text-token rates, so latency is bought with money and portability.
How do you measure a voice latency budget?
Measure the budget by timestamping each stage boundary on every turn and reporting P50 and P95 per stage and for the whole turn. The reference tells teams to instrument each stage separately because bottlenecks are rarely where intuition points, and the evaluating-voice-agents reference names P50 and P95 glass-to-glass percentiles against the sub-600 ms target.
A minimal set of timestamps per turn, as inference about what is practical:
- The moment the caller's speech ended, as your system judged it.
- The moment the final transcript arrived.
- The first language-model token and each tool call's start and end.
- The first synthesized audio chunk, and the moment it began playing.
Report P95 as well as P50 because a caller notices the slow turns. A median inside budget with a P95 far above it is a budget that is not held. Where the clock starts matters too: if the system's judgment of the end of speech runs late, every downstream number looks better than the caller experiences, so define glass to glass from what the caller hears and document how you capture it. Alert on the same percentiles online, using the approach in online agent metrics and drift monitoring.
How far can vendor latency numbers be trusted?
Vendor latency numbers are marketing-adjacent claims to be checked, not inputs to a budget, and the corpus labels them accordingly. The voice-realtime-agents reference verified its vendor entries on 2026-07-09 and states that its 2026-09-28 pass could not re-verify them, so the figures below are vendor-reported as of July 2026.
| Vendor claim | Source in the reference | Caveat |
|---|---|---|
| OpenAI Realtime: roughly 300 to 600 ms glass to glass | Approximate, on subsequent turns | Says nothing about the first turn |
| xAI Grok Voice Agent API: sub-1-second time to first audio | Vendor feature list | A ceiling, not a typical value |
| Cartesia Sonic-3.5: about 40 ms first audio on Turbo, sub-100 ms standard | Vendor-reported, released June 2026 | One stage only, not a whole turn |
Note the units: the OpenAI figure is for a whole turn, the Cartesia figure covers a single stage and the xAI figure is a sub-1-second bound. They are not comparable with each other, and none was measured on phone-network paths or on your region and prompt size.
What can ChangeGamer say from its own data?
ChangeGamer has no measured voice latency, because it runs no voice agent and has taken no call traffic. What it has is demand and ranking data from its 2026-10-01 snapshot (agents/analytics/seo-snapshots/2026-10-01-gh-measure.json): the query "ai voice agent" shows 22,200 searches a month against 720 for "voice ai agent", and the site's only voice rankings are position 62 for "realtime voice ai" and position 71 for "real time voice chat", both on voice and realtime agents. Those figures describe search demand and page positions on one date, not how any voice agent performs. Everything above on latency comes from the reference corpus and the author's arithmetic. The customer-support angle, where a slow answer meets a frustrated caller, is covered in the AI customer support agents reference and the pillar guide.
Frequently asked questions
- What is a good latency for a voice AI agent?
- The reference target in this site's corpus is under 600 ms glass to glass, meaning from the end of the caller's speech to the first audio of the reply. It is a design target for natural-feeling turn-taking rather than a measured industry average, so confirm it with percentiles from your own calls.
- Why does tool calling make a voice agent feel slow?
- A tool call made during a turn happens before the agent can speak its answer, so its duration adds directly to the caller's wait. The voice-realtime-agents reference recommends keeping each mid-conversation tool call under 200 ms, which already takes up a third of a 600 ms budget.
- Is a speech-to-speech model always faster than a cascaded voice pipeline?
- The voice-realtime-agents reference describes native speech-to-speech as lower in end-to-end latency because one model replaces three, but it also records lock-in and audio tokens priced at 3 to 10 times text rates. The latency gap is vendor-reported and unmeasured by this site, so test both on your own audio.
- Should I measure voice agent latency per stage or only end to end?
- Measure both. The end-to-end glass-to-glass number tells you whether the caller waits too long, and per-stage P50 and P95 for transcription, first token and first audio tell you which stage to fix, since the reference warns that bottlenecks are rarely where intuition points.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.