← All resources

How to Choose an LLM for Agentic Tasks

Guide · updated 2026-06-25 · Markdown variant

A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.

The best LLM for an agent is the cheapest one that passes your task-specific evals. Public leaderboards help you build a shortlist; they do not tell you which model works for your tools, your prompts, and your error-recovery patterns. Run your own evals.

Key facts

• Cost, not raw capability, should decide between models that already clear your own agent-specific evaluation bar; treat public leaderboard rank as a shortlisting tool, not the answer.
• Because per-step mistakes compound, a model that is right nine times out of ten on any given step still finishes a ten-step job correctly only about a third of the time.
• Key selection criteria are tool-calling reliability, instruction following, structured-output validity, long-context behavior, latency per step, and cost per full trajectory (not per token).
• BFCL covers function-call accuracy, tau-bench covers reliability across repeated attempts, and SWE-bench Verified covers full coding-task completion — treat all three as a shortlist filter, then validate against your own eval set.
• The recommended workflow scopes the task requirements first, narrows candidates via public benchmarks, sanity-checks a frontier model before optimizing, tests the shortlist on your own evaluation harness, weighs routing cheaper steps to less expensive models, and repeats on a roughly two-month cadence.
• Lean open-weight for high steady call volume, sensitive data, or rate-limit ceilings; lean proxied for low/bursty volume or when the latest weights are needed immediately.

What makes model selection different for agents?

Single-turn LLM quality does not predict agentic performance. Errors compound across steps: 90% per-step accuracy over a 10-step trajectory yields roughly a 35% task completion rate (0.9^10 ≈ 0.35). The failure modes that matter most for agents — tool-call malformation, instruction drift under long context, and poor recovery after a tool error — are not measured well by standard chat benchmarks.

Selection criteria

Which benchmarks to consult

• BFCL (the Berkeley Function-Calling Leaderboard, gorilla.cs.berkeley.edu) — the de-facto standard for function/tool-call correctness. Filter by your target call type rather than the headline number.
• tau-bench (github.com/sierra-research/tau-bench) — measures consistency across repeated i.i.d. trials (pass^k). A model that succeeds 60% individually but rarely succeeds on all k trials is not production-ready.
• SWE-bench Verified (swebench.com/verified.html) — end-to-end coding-agent task completion on human-validated GitHub issues; use it for coding-agent selection. SWE-bench Pro is a contamination-resistant successor for fresher evaluation.
• Your own eval set — the only benchmark that reflects your tools, schemas, prompts, and error-recovery paths. Public leaderboards are a shortlist filter, not the decision.

Step-by-step selection procedure

1. Define your task profile — steps per trajectory, the tool schemas the model must call, input length at p50/p95, acceptable per-step error rate, latency budget, and cost budget per task.
2. Build a shortlist from public benchmarks — use BFCL for your call type, SWE-bench for coding agents, tau-bench pass^k if consistency matters. Do not pick from a benchmark category that does not match your task.
3. Prototype on a frontier model first — get a correctness ceiling and a working eval harness before optimizing. Starting on a small model wastes cycles debugging model limits instead of task logic.
4. Run your shortlist against your evals — replay the same tasks with each model; score task completion, tool-call error rate, total tokens per task, and wall-clock time. Rank by cost per task at your quality threshold, not by per-token price.
5. Consider cascade routing — route simple steps (extraction, classification, formatting) to a cheaper model and reserve the frontier for planning and recovery. Cascade routing can cut cost substantially versus always-frontier while retaining most of the quality (see RouteLLM, arxiv.org/abs/2406.18665, and /resources/ai-gateways-llm-routing).
6. Re-evaluate on a cadence — model quality and pricing change monthly; schedule a lightweight re-eval every 6–8 weeks for production agents. See /resources/evaluating-ai-agents for a harness that makes re-runs cheap.

Open-weight vs proxied: decision heuristic

For open-weight options, see /resources/open-weight-models-for-agents. For cost/latency tuning once a model is chosen, see /resources/agent-cost-latency-optimization. For tool-call reliability mechanisms, see /resources/reliable-tool-calling.

Bottom line

Public benchmarks — BFCL, tau-bench, SWE-bench Verified — only get you a shortlist; they do not measure your tools, your prompts, or your error-recovery paths, so your own eval set is what actually decides. Once a model clears that bar, rank the survivors by cost per full trajectory, not headline per-token price, since a cheaper-per-token model that needs more steps can cost more per task. Layer the open-weight-vs-proxied heuristic on top of that: high steady volume, sensitive data, or rate-limit pressure favor open-weight, while low or bursty volume and the need for the latest weights favor a proxied model. In one line: benchmarks narrow the field, your evals pick the model, and cost-per-task decides between near-ties.

Verified sources

• Berkeley Function-Calling Leaderboard (BFCL): https://gorilla.cs.berkeley.edu/leaderboard.html
• tau-bench (Sierra Research): https://github.com/sierra-research/tau-bench
• SWE-bench Verified: https://www.swebench.com/verified.html
• RouteLLM (cascade routing, arXiv:2406.18665): https://arxiv.org/abs/2406.18665

#llm #agents #model-selection #tool-calling #evaluation #cost #latency #benchmarks

Category: Guide