← All resources

Evaluating AI Agents: Benchmarks and Methods

Reference · updated 2026-06-15 · Markdown variant

Why agent eval differs from single-turn LLM eval, a verified benchmark reference table (SWE-bench, GAIA, BFCL, tau-bench, WebArena, AgentBench, MLE-bench, OSWorld), and practical evaluation methods for agent builders.

Evaluating a multi-step agent is fundamentally different from evaluating a single-turn LLM response. The gap matters: a model that scores 90% on a chat benchmark may fail badly as an agent if it cannot recover from tool errors, maintain state across steps, or complete long-horizon tasks reliably.

Why agent eval differs from single-turn LLM eval

• Multi-step trajectories — correctness at any single step does not imply task completion. Errors compound across steps; partial-credit metrics must account for intermediate progress, not just final outcome.
• Tool use and state — agents call external tools whose outputs are stochastic (network, APIs, environment). Eval must sandbox or mock those calls consistently.
• Non-determinism — the same prompt can produce different tool sequences across runs. A single pass/fail measurement is misleading; multiple runs and reliability metrics (pass^k) are required.
• Cost and latency as first-class metrics — an agent that succeeds but costs 100x more or takes 10x longer than a baseline is not production-ready. Token spend per task and wall-clock time should be tracked alongside accuracy.
• Partial credit — for long tasks, knowing that an agent completed 8 of 10 sub-steps correctly is more useful than a binary pass/fail.

Benchmark reference table

Evaluation methods beyond benchmarks

Trajectory / step-level evaluation

Instead of measuring only final task success, score each intermediate step: did the agent call the right tool? Were arguments correct? Did it recover from an error? Step-level scoring gives partial credit and pinpoints failure modes (bad tool selection vs bad argument generation vs bad recovery logic).

LLM-as-judge

An LLM evaluates another LLM's output using a rubric. Scales better than human annotation for large eval sets. Known failure modes:

• Position bias — the judge systematically favors responses placed first or last in pairwise comparisons. Mitigate by randomizing order and averaging flipped-order results.
• Verbosity bias — judges prefer longer, more formal responses regardless of correctness. Mitigate with rubrics that reward concision and penalize padding.
• Self-preference — a model rates its own outputs higher (lower perplexity → higher score). Mitigate by using a different family as judge.

Use LLM-as-judge for open-ended subjective quality; use exact-match or execution-based eval for tool calls and code.

Rubric / criteria-based grading

Define explicit scoring criteria before running the eval (correctness, safety, tool-use efficiency, instruction following). Apply the rubric consistently — either by a human or an LLM judge prompted with the rubric. Rubric-based grading reduces judge variance and produces auditable scores.

Ground-truth / exact-match for tool calls

For tool-call evaluation, compare the agent's emitted call against a reference: tool name match, argument schema match, and optionally argument value match. BFCL uses AST matching for this. Exact-match is the most reliable method for function-call correctness; it does not require a judge and is not subject to verbosity or position bias.

pass@k vs pass^k

• pass@k — probability at least one of k independent samples succeeds. Common in code generation; measures ceiling capability.
• pass^k — probability all k i.i.d. trials succeed (tau-bench's primary metric). Measures reliability and consistency. An agent that passes 50% of trials is very different from one that always succeeds on half the tasks and always fails on the other half — pass^k surfaces this variance.

Online vs offline eval

• Offline — evaluate against a fixed dataset with pre-computed reference outputs. Fast, reproducible, cheap. Risk: the agent may have seen the data during training (benchmark contamination).
• Online — evaluate on live tasks drawn from real user interactions or a live environment. Catches distribution shift and contamination. Slower and harder to reproduce.

Human eval

The ground truth for open-ended tasks. Use for calibrating automated evals, not as the primary scalable signal. Human baseline scores on GAIA (~92%), WebArena (~78%), and VisualWebArena (~88.7%) give the ceiling most public benchmarks are still far below.

Practical guidance

• Eval your own task distribution, not just public benchmarks. Public benchmarks measure proxy tasks. The gap between benchmark score and production performance is typically large.
• Measure cost and latency per task. An agent that solves a task in 2 tool calls and $0.01 is better than one that solves it in 20 calls and $1.00, even if both score 100%.
• Track tool-call accuracy. For production agents, tool-call precision is often more predictive of reliability than end-task success rate. Use BFCL and your own ground-truth tool-call logs. See /resources/reliable-tool-calling for mechanisms and failure modes.
• Guard against benchmark contamination. Models trained after a benchmark was released may have seen its tasks. Prefer held-out or dynamically generated eval sets for final production assessment. SWE-bench Verified was deprecated by OpenAI in February 2026 partly due to contamination concerns.
• Check framework-native eval and observability tooling. LangGraph ships LangSmith, Pydantic AI ships Logfire, OpenAI Agents SDK has built-in tracing. Framework-level traces are the fastest path to step-level eval. See /resources/agent-frameworks-compared.

Verified sources

• SWE-bench GitHub (Princeton NLP / SWE-bench org): https://github.com/swe-bench/SWE-bench
• SWE-bench Verified introduction (OpenAI): https://openai.com/index/introducing-swe-bench-verified/
• GAIA paper (Meta AI + HuggingFace, 2023): https://huggingface.co/papers/2311.12983
• GAIA leaderboard (HuggingFace): https://huggingface.co/learn/agents-course/unit4/what-is-gaia
• BFCL paper (ICML 2025): https://proceedings.mlr.press/v267/patil25a.html
• BFCL GitHub (Gorilla / UC Berkeley): https://github.com/ShishirPatil/gorilla/blob/main/berkeley-function-call-leaderboard/README.md
• tau-bench (Sierra Research): https://github.com/sierra-research/tau-bench
• tau-bench paper (arXiv 2406.12045): https://arxiv.org/abs/2406.12045
• tau2-bench (Sierra Research): https://github.com/sierra-research/tau2-bench
• WebArena GitHub (CMU / web-arena-x): https://github.com/web-arena-x/webarena
• VisualWebArena paper (CMU, ACL 2024): https://arxiv.org/abs/2401.13649
• AgentBench paper (THUDM / Tsinghua, ICLR 2024): https://arxiv.org/abs/2308.03688
• AgentBench GitHub: https://github.com/THUDM/AgentBench
• MLE-bench (OpenAI, ICLR 2025): https://openai.com/index/mle-bench/
• MLE-bench GitHub: https://github.com/openai/mle-bench
• OSWorld benchmark site: https://os-world.github.io/

#evaluation #benchmarks #agents #evals #tool-calling #trajectory

Category: Reference