#evaluation

8 resources and 9 guides tagged #evaluation on ChangeGamer.

AI Agent Evaluation: Benchmarks and Methods · Reference
Why agent eval differs from single-turn LLM eval, a verified benchmark reference table (SWE-bench, GAIA, BFCL, tau-bench, WebArena, AgentBench, MLE-bench, OSWorld), and practical evaluation methods for agent builders.
Evaluating Voice Agents: Metrics and Benchmarks · Reference
How to score a voice agent per pipeline stage — WER for STT, MOS for TTS, VoiceBench and τ³-bench for end-to-end behavior — layered on top of general agent-eval methods.
Agent Observability and Tracing · Guide
Why agents need observability beyond app logs, how OpenTelemetry GenAI semantic conventions model agent runs as traces, key signals to capture, and a verified tooling landscape.
RAG and Retrieval for Agents · Guide
End-to-end practitioner reference for Retrieval-Augmented Generation: pipeline stages, chunking strategies, dense/sparse/hybrid retrieval, reranking, agentic retrieval patterns, quality failure modes, and evaluation — with verified sources for every named technique.
Shipping AI Agents to Production: A Production-Readiness Checklist · Guide
End-to-end checklist for productionizing an AI agent — evaluation gates, observability, guardrails, cost controls, resilience, durability, HITL approvals, secrets, rollback, and incident response.
Reranking for RAG: Cross-Encoders, LLM Rerankers, and Hosted APIs · Guide
Second-stage retrieval step that re-scores bi-encoder candidates with full query-document attention, boosting precision without sacrificing recall; covers cross-encoder, LLM, and late-interaction reranking, hosted APIs, sizing heuristics, and evaluation.
How to Choose an LLM for Agentic Tasks · Guide
A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.
Prompt Management and Versioning: The Ops of Prompts in Production · Guide
Treating prompts as deployable artifacts: versioning, external registries, A/B and canary testing, eval-gated promotion, rollback, and the composite-version problem.

Guides

Agentic RAG in Production: The Complete Operator Guide · pillar guide
The operator playbook for agentic RAG in production: ingestion, chunking, hybrid retrieval, reranking, evaluation, freshness and cost.
How to Evaluate a RAG System (Retrieval Metrics, Generation Metrics, CI Gates) · guide
The evaluation harness that keeps RAG changeable: golden-set construction, retrieval metrics separated from generation metrics, LLM-as-judge screening with human acceptance, CI regression gates, and the logged-query flywheel.
Agent Guardrails and the AI Agent Reliability Playbook · pillar guide
Agent guardrails plus the eleven other disciplines that make an AI agent reliable in production: tool calling, retries, durable execution and rollout.
How to Evaluate AI Agents in CI · guide
An operator playbook for gating an AI agent release in CI: why agent eval needs trajectory-level scoring across the tasks it actually runs, how public benchmarks diverge as proxies, ground-truth vs LLM-as-judge tool-call scoring, and the three-layer test pyramid that keeps CI fast and non-flaky.
AI Agent Observability and the Production Evaluation Playbook · pillar guide
AI agent observability and evaluation once an agent is already live: tracing spans, retrieval-attribution logs, judge-based screening vs. human acceptance, online metrics, cost telemetry, and incident postmortems that feed evaluation fixtures.
Designing an LLM-as-Judge Pipeline for Production AI Agents · guide
An operator playbook for screening live AI agent output with an LLM judge: a confidence/stakes routing architecture to a human queue, continuous live-traffic rubric design, and per-bias mitigations for position, verbosity, and self-preference.
How to Turn an AI Agent Incident into an Evaluation Test Case · guide
How to turn an AI agent incident into an evaluation test case: freeze the trace, redact it, minimize it to the failing decision, label the expected outcome with a trusted oracle, prove it fails then passes, and retire it later.
How to Measure AI Agent Quality from Live User Feedback · guide
How to measure AI agent quality from live user feedback: why explicit ratings are sparse and biased, how re-asks, abandonment and escalations mislead, and how to join each signal to a trace and route it to review.
How to Detect Quality Drift in a Production AI Agent · guide
How to detect quality drift in a production AI agent: baseline aggregate signals, alert on a diff against the baseline, and separate a prompt, model or tool-version change from a shift in traffic mix.