← All tags · All resources · All guides

#latency

6 resources and 2 guides tagged #latency on ChangeGamer.

• Evaluating Voice Agents: Metrics and Benchmarks · Reference
  How to score a voice agent per pipeline stage — WER for STT, MOS for TTS, VoiceBench and τ³-bench for end-to-end behavior — layered on top of general agent-eval methods.
• Agent Cost and Latency Optimization · Guide
  Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).
• LLM Streaming for Agents: Server-Sent Events and Provider Event Formats · Guide
  Transport formats, provider event schemas, and practical concerns for consuming streamed LLM responses in production agents: SSE mechanics, OpenAI (Chat Completions and Responses API) and Anthropic event formats, partial-JSON tool-call parsing, backpressure, cancellation, and gateway proxying.
• Prompt Caching for AI Agents · Guide
  Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.
• How to Choose an LLM for Agentic Tasks · Guide
  A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.
• Application-Level Response Caching for AI Agents · Guide
  How to implement exact-match and semantic caching in your agent application to eliminate redundant LLM calls, with threshold guidance, invalidation strategies, and a decision matrix for when semantic caching is unsafe.

Guides

• When and How to Rerank Retrieved Documents in RAG · guide
  Reranking as a budget decision: why first-stage ranking misorders good evidence, when cross-encoder reranking pays for itself, how to pick candidate depth at the knee, gating by query difficulty, and deduplicating after fusion.
• How to Cut RAG Cost and Latency Without Cutting Quality · guide
  RAG cost and latency as engineered budgets: where the money actually goes, caching layers and their hit-rate economics, routing queries to right-sized models, bounding retrieval fan-out, the hidden lines (reindex migrations, eval compute), and p95 discipline.