#latency
6 resources and 6 guides tagged #latency on ChangeGamer. Includes Evaluating Voice Agents: Metrics and Benchmarks and Agent Cost and Latency Optimization.
- Evaluating Voice Agents: Metrics and Benchmarks How to score a voice agent per pipeline stage — WER for STT, MOS for TTS, VoiceBench and τ³-bench for end-to-end behavior — layered on top of general agent-eval methods.
- Agent Cost and Latency Optimization Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).
- LLM Streaming for Agents: Server-Sent Events and Provider Event Formats Transport formats, provider event schemas, and practical concerns for consuming streamed LLM responses in production agents: SSE mechanics, OpenAI (Chat Completions and Responses API) and Anthropic event formats, partial-JSON tool-call parsing, backpressure, cancellation, and gateway proxying.
- Prompt Caching for AI Agents Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.
- How to Choose an LLM for Agentic Tasks A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.
- Application-Level Response Caching for AI Agents How to implement exact-match and semantic caching in your agent application to eliminate redundant LLM calls, with threshold guidance, invalidation strategies, and a decision matrix for when semantic caching is unsafe.
Guides
- When and How to Rerank Retrieved Documents in RAG Reranking as a budget decision: why first-stage ranking misorders good evidence, when cross-encoder reranking pays for itself, how to pick candidate depth at the knee, gating by query difficulty, and deduplicating after fusion.
- How to Cut RAG Cost and Latency Without Cutting Quality RAG cost and latency as engineered budgets: where the money actually goes, caching layers and their hit-rate economics, routing queries to right-sized models, bounding retrieval fan-out, the hidden lines (reindex migrations, eval compute), and p95 discipline.
- Voice AI Agent in Production: Architecture, Latency, Handoff and Launch How to build and run a voice AI agent in production: cascaded vs speech-to-speech architecture, turn detection and barge-in, the latency budget, tool calls in a voice loop, transport, cost, human handoff and launch sequencing.
- Voice Agent Latency Budget: Where the 600 ms Goes and How to Hold It How to split a 600 ms voice agent latency budget across speech-to-text, the language model, speech synthesis and tool calls, how to measure each stage, and which levers hold the budget.
- How to Monitor a Voice Agent in Production: Dashboard, Alerts and Runbook How to monitor a voice agent in production: per-stage latency percentiles, transcription confidence, containment paired with a resolution signal, live false-interruption rate, a handoff trigger review, and call-audio retention, each with what to chart, alert on and do.
- Browser Agent Cost per Task: A Formula, the Levers and the Latency Caveat How to compute browser agent cost per task as of 14 November 2026: a formula for screenshots and toolset overhead, an illustrative worked example, Anthropic image-token sizes, the cost levers and why the vendor pages read on 14 November 2026 give no latency figure.