#evaluation
8 resources and 17 guides tagged #evaluation on ChangeGamer. Includes AI Agent Evaluation: Benchmarks and Methods.
- AI Agent Evaluation: Benchmarks and Methods Why agent eval differs from single-turn LLM eval, a verified benchmark reference table (SWE-bench, GAIA, BFCL, tau-bench, WebArena, AgentBench, MLE-bench, OSWorld), and practical evaluation methods for agent builders.
- Evaluating Voice Agents: Metrics and Benchmarks How to score a voice agent per pipeline stage — WER for STT, MOS for TTS, VoiceBench and τ³-bench for end-to-end behavior — layered on top of general agent-eval methods.
- Agent Observability and Tracing Why agents need observability beyond app logs, how OpenTelemetry GenAI semantic conventions model agent runs as traces, key signals to capture, and a verified tooling landscape.
- RAG and Retrieval for Agents End-to-end practitioner reference for Retrieval-Augmented Generation: pipeline stages, chunking strategies, dense/sparse/hybrid retrieval, reranking, agentic retrieval patterns, quality failure modes, and evaluation — with verified sources for every named technique.
- Shipping AI Agents to Production: A Production-Readiness Checklist End-to-end checklist for productionizing an AI agent — evaluation gates, observability, guardrails, cost controls, resilience, durability, HITL approvals, secrets, rollback, and incident response.
- Reranking for RAG: Cross-Encoders, LLM Rerankers, and Hosted APIs Second-stage retrieval step that re-scores bi-encoder candidates with full query-document attention, boosting precision without sacrificing recall; covers cross-encoder, LLM, and late-interaction reranking, hosted APIs, sizing heuristics, and evaluation.
- How to Choose an LLM for Agentic Tasks A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.
- Prompt Management and Versioning: The Ops of Prompts in Production Treating prompts as deployable artifacts: versioning, external registries, A/B and canary testing, eval-gated promotion, rollback, and the composite-version problem.
Guides
- Agentic RAG in Production: The Complete Operator Guide The operator playbook for agentic RAG in production: ingestion, chunking, hybrid retrieval, reranking, evaluation, freshness and cost.
- How to Evaluate a RAG System (Retrieval Metrics, Generation Metrics, CI Gates) The evaluation harness that keeps RAG changeable: golden-set construction, retrieval metrics separated from generation metrics, LLM-as-judge screening with human acceptance, CI regression gates, and the logged-query flywheel.
- Agent Guardrails and the AI Agent Reliability Playbook Agent guardrails plus the eleven other disciplines that make an AI agent reliable in production: tool calling, retries, durable execution and rollout.
- How to Evaluate AI Agents in CI An operator playbook for gating an AI agent release in CI: why agent eval needs trajectory-level scoring across the tasks it actually runs, how public benchmarks diverge as proxies, ground-truth vs LLM-as-judge tool-call scoring, and the three-layer test pyramid that keeps CI fast and non-flaky.
- AI Agent Observability and the Production Evaluation Playbook AI agent observability and evaluation once an agent is already live: tracing spans, retrieval-attribution logs, judge-based screening vs. human acceptance, online metrics, cost telemetry, and incident postmortems that feed evaluation fixtures.
- Designing an LLM-as-Judge Pipeline for Production AI Agents An operator playbook for screening live AI agent output with an LLM judge: a confidence/stakes routing architecture to a human queue, continuous live-traffic rubric design, and per-bias mitigations for position, verbosity, and self-preference.
- How to Turn an AI Agent Incident into an Evaluation Test Case How to turn an AI agent incident into an evaluation test case: freeze the trace, redact it, minimize it to the failing decision, label the expected outcome with a trusted oracle, prove it fails then passes, and retire it later.
- How to Measure AI Agent Quality from Live User Feedback How to measure AI agent quality from live user feedback: why explicit ratings are sparse and biased, how re-asks, abandonment and escalations mislead, and how to join each signal to a trace and route it to review.
- How to Detect Quality Drift in a Production AI Agent How to detect quality drift in a production AI agent: baseline aggregate signals, alert on a diff against the baseline, and separate a prompt, model or tool-version change from a shift in traffic mix.
- How to Test Voice Agent Turn Detection and Barge-In How to test voice agent turn detection and barge-in: build a labelled clip set, count barge-in accuracy and false-interruption rate, separate cut-off from talk-over failures, and split offline gates from online monitors.
- How to Handle Voice AI Vendor Model Changes Without Regressing Callers How to handle voice AI vendor model changes: classify the change, record the model id on every trace, replay an offline set with hard calls on old and new, canary before cutover, keep a rollback, and run a deprecation-readiness routine.
- How Does a Voice Agent Handle Callers Who Speak Another Language? How a live voice agent handles callers who speak another language: detecting the language, choosing speech components per language, slicing evaluation by language and handing off when the language is unsupported, with the thin corpus evidence stated plainly.
- Measuring the Quality of an AI Customer Support Agent: Resolution Signals Beside Deflection How to measure the quality of an AI customer support agent: explicit confirmation, a no-repeat-contact window and a post-resolution survey, each with its blind spot, read together with the escalation log and a verdict table.
- How to Migrate From a Rules-Based Support Chatbot to an AI Support Agent A migration sequence from a decision-tree support bot to a model-driven agent: inventory intents and escalation exits, shadow mode, per-intent cutover, the tree as fallback, parity measurement and rollback triggers.
- How to Set Up a Human Review Queue for an AI Support Agent: Sample, Sign Off, Cap the Load How to set up a human review queue for an AI support agent: what to sample, who reviews, a four-outcome decision set, a tie-break for disagreements, fixtures from corrections, and a cap on reviewer load.
- How to Test an AI Support Agent With Historical Tickets: Build a Replay Set How to test an AI support agent with historical tickets: pick resolved tickets per request type including escalations and refusals, redact them, assert on behaviour not wording, grade deterministically first, bridge to live traffic with shadow mode, and gate releases.
- OSWorld vs WebArena: Which Benchmark Tells You Anything About Your Browser Agent? OSWorld vs WebArena as read on 20 November 2026: what OSWorld, OSWorld-Verified, OSWorld 2.0, WebArena, VisualWebArena and Online-Mind2Web each measure, how they are scored, what running them involves and what none of them covers.