← All guides · Voice agents in production

How to Monitor a Voice Agent in Production: Dashboard, Alerts and Runbook

Part 9 of Voice agents in production · 1,305 words · ~6 min read · published 2026-10-12 · updated 2026-10-12 · Markdown variant

How to monitor a voice agent in production: per-stage latency percentiles, transcription confidence, containment paired with a resolution signal, live false-interruption rate, a handoff trigger review, and call-audio retention, each with what to chart, alert on and do.

Part of the Voice AI Agent in Production: Architecture, Latency, Handoff and Launch guide.

A voice agent needs a live dashboard, a short list of alerts and a runbook that says what to do when each one fires; as inference, the failures are audible to callers before a text error log shows anything. This article is the operator's version of the monitoring half of the voice AI agent in production pillar. Metric definitions live in the evaluating voice agents reference and are not restated here; the question here is what to put on a screen and what to do about it.

Labels: "corpus" is a claim from a named reference, "inference" is the author's reasoning. The corpus names the signals but gives no alert levels, so every threshold below is "set from your own baseline".

What belongs on a voice agent dashboard?

A voice agent dashboard needs five panels: latency by stage, transcription confidence, containment next to a resolution signal, false interruptions, and handoff triggers. The first, second and third panels come from the online signals the evaluating-voice-agents reference lists (corpus); the panel layout is inference.

Tie each alert to a named person and a first step, or delete it. An alert nobody can act on trains the team to ignore the next one.

How do you alert on per-stage latency percentiles?

Alert when any stage drifts from its own baseline or when the total breaches the budget; the split and the timestamping method are in the voice agent latency budget, and the per-stage alerting rule is inference.

• Chart: each stage and the total, using that article's percentiles.
• Alert on: a stage whose P95 moves away from its own recent baseline even if the total still passes, plus any sustained breach of the total.
• Action: open a sampled slow call, read its trace to find the stage, then check the usual suspects: a tool call that stalled, a prompt that grew, or a vendor change. The agent-observability reference describes the trace-and-span model and says to attach exceptions to the span that failed, which is how a slow stage becomes findable.

As inference, charting only the total can hide a drifting stage, since stage changes may offset each other. For generic drift mechanics see online agent metrics and drift monitoring.

What do you do when transcription confidence drops?

Treat a shift in the transcription confidence distribution as an early warning that the audio, not the model, has changed, and respond by sampling the low-confidence calls. The reference lists transcription confidence as an online signal (corpus); what a drop means is inference.

• Chart: the distribution of per-call confidence, split by whatever you can observe cheaply, such as telephony route or region.
• Alert on: a change in the distribution's shape, not a single low call.
• Action: listen to a sample from the affected slice. A route-specific drop points at the audio path; a drop everywhere after a date points at a vendor or model change.

Transcription confidence is the score your speech-to-text provider exposes; the corpus does not define it, so compare it only within one vendor and model version (inference). A mis-heard value that reaches a tool call is more costly than a mis-heard filler word, so also chart how often critical values such as order numbers were confirmed back, a design covered in voice agent guardrails for spoken output.

How do you read containment without fooling yourself?

Read containment only next to a resolution signal, because a contained call includes callers who quit. The customer-support reference, as quoted in the pillar, cautions that a deflection count cannot separate resolution from abandonment (corpus); the pairing is the remedy.

• Chart: containment and a repeat-contact rate for the same caller or topic on one shared time axis. If you cannot link repeat calls, chart a weekly sample of ended calls graded for whether the issue was resolved.
• Alert on: containment rising while repeat contact also rises, or while sampled resolution falls.
• Action: read the sampled calls that ended without a handoff. Check whether callers hung up mid-task, and whether a refusal or loop pushed them out.

The grade of a sampled call is a judgment, so write the rubric down once and reuse it, or the trend measures the grader. Judge-style rubrics are discussed in agent observability and the production evaluation playbook, but writing the rubric for your own sampled calls is a step that playbook does not spell out.

How do you watch false interruptions on live calls?

Chart false-interruption rate daily and alert on a step change, because a sudden jump usually follows a configuration, vendor or audio-path change. The reference names false-interruption rate and barge-in accuracy among end-to-end measures (corpus); the live version is inference.

• Chart: the rate per day, plus count of agent turns cut off early in the utterance if your logs can show it (define the window yourself, inference).
• Alert on: a step change aligned with a deploy date, not slow noise.
• Action: replay affected calls through the test method in how to test turn detection and barge-in, then compare the failing audio with your offline set.

Live data carries no ground truth on what the caller meant, so a person has to label a sample before the number is trusted. A rising rate with no deploy suggests the callers or their devices changed; add those clips to the offline set.

How do you review the handoff trigger log?

Review the handoff trigger log weekly by category, because a new or growing category reveals a gap in the agent faster than any aggregate. Log contents and transfer mechanics are in voice agent human handoff; only the review rules follow.

1. Chart handoffs by trigger category.
2. Alert on a category appearing for the first time or taking a growing share, and on any handoff with a missing payload field.
3. Act by reading a handful of calls in that category and deciding whether to fix the agent, change the trigger, or accept it.

Treat a falling handoff count with suspicion until the resolution panel agrees. Fewer transfers can mean the agent is better, or that callers can no longer reach a person.

How long should call audio be kept and how should it be redacted?

Keep call audio only for sampled or flagged calls, for a retention period you state in writing, with redaction applied before anyone reviews it. That is inference; the pillar says to decide retention before launch, and the redaction method is in redacting sensitive data from agent traces.

What does ChangeGamer know first-hand about monitoring a voice agent?

Nothing about operating one: as of October 2026 ChangeGamer runs no voice agent and has no call data, so no figure above is measured. The one verifiable fact is about its own build: node scripts/verify-invariants.mjs, run in changegamer/ after npm run build, prints "All invariants hold." when its checks pass, and an alert that can be checked as mechanically as that one is the model to copy: it passes or fails with no judgment call. For the full decision guide, return to the voice agents cluster, and for the realtime vendor landscape see the voice and realtime agents reference.

#voice #monitoring #observability #alerting #latency #agents #production