What to Log and Filter in a Voice Agent, by Architecture
What to log and filter in a voice agent depending on architecture: where a cascaded pipeline gives you text seams for logging, redaction and output filters, what a native speech-to-speech model leaves you to do from outside, and what a model swap forces you to re-test.
- As the author's inference from the reference, a cascaded voice agent exposes text between hearing and speaking where a team can log, redact or filter, while a native speech-to-speech model has no text intermediate at its core inference step.
- The voice-realtime-agents reference does not state what logging or filtering a native speech-to-speech API supports, so a team should read each vendor's documentation for returned transcripts and events before designing around them.
- The evaluating-voice-agents reference does not say which of its per-stage metrics, such as word error rate or mean opinion score, apply to a native speech-to-speech model, so only whole-turn measures can be assumed.
- Swapping a model or vendor in a voice agent should trigger a re-run of the same labelled call set, because a change in one component can alter behavior that the other components appear to own.
- As inference, a prototype need not lock in the final architecture, since the voice-realtime-agents reference says LiveKit Agents covers both backends without application code changes, a sentence verified on 2026-07-09 and not separately re-checked on 2026-09-28.
What you can log and filter in a voice agent depends on its architecture, because a cascaded pipeline hands text from one model to the next and a native speech-to-speech model does not. This article covers the seams, what to record at each, and what a component swap forces you to re-test. Which architecture to choose is the pillar's question, answered in cascaded pipeline or native speech-to-speech: which architecture should you choose?. Delay numbers are in the voice agent latency budget and interruption testing is in how to test turn detection and barge-in. The sources are voice and realtime agents and evaluating voice agents.
Where can you log, redact or filter between hearing and speaking?
As the author's inference, a cascaded pipeline lets you act on the transcript between speech-to-text and the language model and again on the reply text before synthesis, whereas a native speech-to-speech model has no text hand-off at its core inference step to act on. The voice-realtime-agents reference (updated 2026-09-28) describes the cascade as three models in sequence and the native model as ingesting and emitting raw audio "without a text intermediate at the core inference step". It does not say what that means for logging or filtering, so the table below is inference too.
| Control point | Cascaded | Native speech-to-speech |
|---|---|---|
| Transcript of what the caller said | Available as a stage output | Needs a separate transcription path, if the vendor offers one |
| Redaction before the model sees it | Possible on the transcript | No text seam inside the model |
| Filter on the reply before it is spoken | Possible on reply text | No text seam inside the model |
| Per-stage logging | One record per stage | One model, one boundary |
Check each vendor's documentation for what its realtime API returns alongside audio, because the corpus does not list that.
What should you log at each seam of a cascaded pipeline?
At each seam of a cascade, log what entered, what left and whether your own code changed it. This is the author's suggestion, since the corpus prescribes no log format.
- After transcription: the raw transcript, the transcript after redaction, and the transcription confidence where the vendor exposes it. The evaluating-voice-agents reference names STT confidence as an online signal, which is a reason to keep it.
- After the language model: the reply text before any filter, any tool calls requested and their results.
- After the output filter: the text actually sent to synthesis, plus a flag when the filter altered or blocked the reply.
- Around synthesis and playback: when audio started, and whether the pending audio was flushed by an interruption.
Keep the pre-redaction and post-redaction versions under different access rules, or do not retain the raw one at all; the reference gives no retention guidance, so that is a decision for your own policy and counsel.
What can you log and filter on a native speech-to-speech model?
On a native model you can reliably log only what you send and receive, meaning the input audio, the output audio and whatever events or transcripts the vendor's API chooses to return. That is inference; the corpus lists vendor features such as tool calling, turn detection and barge-in for the realtime APIs but not what transcripts or events each returns.
Practical consequences, as the author's reasoning:
- Redaction has to happen before the audio reaches the model, if it can happen at all, because there is no text stage to scrub afterwards.
- Output filtering has to rely on the model's instructions and on limiting what its tools can do, with any transcript the vendor returns used for after-the-fact review rather than blocking.
- Evaluation relies on recorded audio replayed through the model and judged at the caller's ear, which is why the labelled call set matters more here.
Which evaluation metrics apply to each architecture?
The evaluating-voice-agents reference (updated 2026-09-15) names per-stage metrics for pipelines and whole-turn measures, but it does not state which of them apply to a native speech-to-speech model. The per-stage list is in the pillar guide and the metric definitions are in evaluating voice agents. As inference, word error rate and mean opinion score assume a transcription stage and a synthesis stage you can score separately, so for a native model only whole-turn measures can safely be assumed unless you transcribe its input and output audio yourself. The reference states no such procedure.
What does swapping a model or vendor force you to re-test?
Swapping any component should trigger a re-run of the full labelled call set, not only the tests for the component you changed. This is inference; the corpus gives no swap procedure. The reasoning is that an upstream change shifts what downstream stages receive, and turn-taking behavior emerges from several components together.
- Swap the transcription model: re-score word error rate on the held-out set, then re-run end-to-end task completion, since transcripts feed everything after them.
- Swap the language model: re-run task completion, tool-call correctness and every output filter, and check that replies still read well when spoken aloud.
- Swap the synthesis voice: re-score intelligibility on names, numbers and acronyms, and recheck barge-in, since the stop-speaking flush acts on that stage's buffer.
- Swap to a different speech-to-speech model: re-run everything end to end, because the model owns turn detection and interruption behavior internally and a version change can move both.
The reference notes that τ³-bench scores its voice mode with pass^k, so replaying the set several times exposes inconsistency. The same reference records an OpenAI model update of 6 July 2026 that mentions improved interruption behavior (vendor-reported as of July 2026, not re-verified 2026-09-28), which is a reason to re-run the interruption set on every model version change.
If the application sits behind LiveKit Agents, the reference says its AgentSession orchestrator covers both backends without application code changes. The 2026-09-28 pass re-checked LiveKit's version and AgentSession as current API but did not separately re-check that cross-backend sentence, which was verified on 2026-07-09. As inference, code portability does not carry over the seam-specific logs and filters above; those are rebuilt or dropped when the architecture changes.
What does ChangeGamer know first-hand about voice architecture?
ChangeGamer runs no voice agent as of October 2026, so nothing in this article was measured from calls. Every behavioral claim comes from the voice-realtime-agents and evaluating-voice-agents references or is marked as the author's inference. The corpus does not state which evaluation metrics apply to native speech-to-speech models, what transcripts or events such APIs return, or how long a model swap takes to re-test. Treat those three as open questions to settle with a vendor and your own recorded audio.
Frequently asked questions
- Can you add guardrails to a native speech-to-speech voice agent?
- The voice-realtime-agents reference describes interruption handling and turn detection as living inside a native speech-to-speech model, and it does not describe an intermediate text point for filtering. As inference, guardrails on such a model therefore sit outside it, on the input audio path or on what the model is allowed to do.
- Which evaluation metrics apply to a native speech-to-speech voice agent?
- The evaluating-voice-agents reference does not say. It gives word error rate and mean opinion score as per-stage metrics for a pipeline with separate transcription and synthesis stages, and whole-turn measures such as latency percentiles, barge-in accuracy and false-interruption rate, so only the whole-turn measures can safely be assumed for a native model.
- Is it hard to switch a voice agent between cascaded and speech-to-speech backends?
- The voice-realtime-agents reference says LiveKit Agents covers cascaded and native backends without changing application code when switching between them. Its 2026-09-28 pass re-checked the framework version and the AgentSession API as current but did not separately re-check that sentence, which was verified on 2026-07-09; re-testing the behavior is a separate cost.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.