# How to Choose an LLM for Agentic Tasks

> A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.

Category: Guide · Updated: 2026-06-25 · Tags: llm, agents, model-selection, tool-calling, evaluation, cost, latency, benchmarks
Canonical: https://changegamer.ai/resources/choosing-an-llm-for-agents
Variants: [HTML](https://changegamer.ai/resources/choosing-an-llm-for-agents) · [JSON](https://changegamer.ai/api/resources/choosing-an-llm-for-agents.json)
License: https://changegamer.ai/license.xml · Access: free

The best LLM for an agent is the cheapest one that passes your task-specific evals. Public leaderboards help you build a shortlist; they do not tell you which model works for your tools, your prompts, and your error-recovery patterns. Run your own evals.

## Key facts

- Cost, not raw capability, should decide between models that already clear your own agent-specific evaluation bar; treat public leaderboard rank as a shortlisting tool, not the answer.
- Because per-step mistakes compound, a model that is right nine times out of ten on any given step still finishes a ten-step job correctly only about a third of the time.
- Key selection criteria are tool-calling reliability, instruction following, structured-output validity, long-context behavior, latency per step, and cost per full trajectory (not per token).
- BFCL covers function-call accuracy, tau-bench covers reliability across repeated attempts, and SWE-bench Verified covers full coding-task completion — treat all three as a shortlist filter, then validate against your own eval set.
- The recommended workflow scopes the task requirements first, narrows candidates via public benchmarks, sanity-checks a frontier model before optimizing, tests the shortlist on your own evaluation harness, weighs routing cheaper steps to less expensive models, and repeats on a roughly two-month cadence.
- Lean open-weight for high steady call volume, sensitive data, or rate-limit ceilings; lean proxied for low/bursty volume or when the latest weights are needed immediately.

## What makes model selection different for agents?

Single-turn LLM quality does not predict agentic performance. Errors compound across steps: 90% per-step accuracy over a 10-step trajectory yields roughly a 35% task completion rate (0.9^10 ≈ 0.35). The failure modes that matter most for agents — tool-call malformation, instruction drift under long context, and poor recovery after a tool error — are not measured well by standard chat benchmarks.

## Selection criteria

| Criterion | What to measure | Why it matters for agents |
|---|---|---|
| **Tool-calling reliability** | Schema-valid call rate on BFCL for your call type (single, parallel, multi-turn) | One malformed call breaks the step; downstream steps operate on corrupt state |
| **Instruction following** | Pass rate on YOUR eval set; IFEval as a proxy | Agents chain instructions; drift accumulates across turns |
| **Structured output** | Strict JSON Schema pass rate under constrained decoding | Downstream parsers treat schema violations as task failures |
| **Long-context behavior** | Retrieval accuracy near the nominal context limit | Nominal window != usable window; quality degrades before the limit |
| **Latency per step** | Median time-to-first-token + completion at your p50 input | Serial loops multiply per-step latency over many steps |
| **Cost per task** | Total tokens × price across a full trajectory, not the per-token headline | A cheaper per-token model that needs more steps can cost more per task |
| **Open-weight vs proxied** | Licensing, self-hosting GPU cost, rate-limit ceiling | Open-weight removes per-call fees and rate ceilings; adds hosting ops |

## Which benchmarks to consult

- **BFCL** (the Berkeley Function-Calling Leaderboard, gorilla.cs.berkeley.edu) — the de-facto standard for function/tool-call correctness. Filter by your target call type rather than the headline number.
- **tau-bench** (github.com/sierra-research/tau-bench) — measures consistency across repeated i.i.d. trials (pass^k). A model that succeeds 60% individually but rarely succeeds on all k trials is not production-ready.
- **SWE-bench Verified** (swebench.com/verified.html) — end-to-end coding-agent task completion on human-validated GitHub issues; use it for coding-agent selection. SWE-bench Pro is a contamination-resistant successor for fresher evaluation.
- **Your own eval set** — the only benchmark that reflects your tools, schemas, prompts, and error-recovery paths. Public leaderboards are a shortlist filter, not the decision.

## Step-by-step selection procedure

1. **Define your task profile** — steps per trajectory, the tool schemas the model must call, input length at p50/p95, acceptable per-step error rate, latency budget, and cost budget per task.
2. **Build a shortlist from public benchmarks** — use BFCL for your call type, SWE-bench for coding agents, tau-bench pass^k if consistency matters. Do not pick from a benchmark category that does not match your task.
3. **Prototype on a frontier model first** — get a correctness ceiling and a working eval harness before optimizing. Starting on a small model wastes cycles debugging model limits instead of task logic.
4. **Run your shortlist against your evals** — replay the same tasks with each model; score task completion, tool-call error rate, total tokens per task, and wall-clock time. Rank by cost per task at your quality threshold, not by per-token price.
5. **Consider cascade routing** — route simple steps (extraction, classification, formatting) to a cheaper model and reserve the frontier for planning and recovery. Cascade routing can cut cost substantially versus always-frontier while retaining most of the quality (see RouteLLM, arxiv.org/abs/2406.18665, and /resources/ai-gateways-llm-routing).
6. **Re-evaluate on a cadence** — model quality and pricing change monthly; schedule a lightweight re-eval every 6–8 weeks for production agents. See /resources/evaluating-ai-agents for a harness that makes re-runs cheap.

## Open-weight vs proxied: decision heuristic

| Signal | Lean open-weight | Lean proxied |
|---|---|---|
| Call volume | High and steady | Low or bursty |
| Data sensitivity | PII / regulated / cannot leave your infra | Public or low-sensitivity |
| Rate limits | Hitting API ceilings | Within tier limits |
| Ops capacity | GPU infra + model-update workflow available | No infra team |
| Model recency | Willing to lag 1–2 versions | Need latest weights immediately |

For open-weight options, see /resources/open-weight-models-for-agents. For cost/latency tuning once a model is chosen, see /resources/agent-cost-latency-optimization. For tool-call reliability mechanisms, see /resources/reliable-tool-calling.

## Bottom line

Public benchmarks — BFCL, tau-bench, SWE-bench Verified — only get you a shortlist; they do not measure your tools, your prompts, or your error-recovery paths, so your own eval set is what actually decides. Once a model clears that bar, rank the survivors by cost per full trajectory, not headline per-token price, since a cheaper-per-token model that needs more steps can cost more per task. Layer the open-weight-vs-proxied heuristic on top of that: high steady volume, sensitive data, or rate-limit pressure favor open-weight, while low or bursty volume and the need for the latest weights favor a proxied model. In one line: benchmarks narrow the field, your evals pick the model, and cost-per-task decides between near-ties.

## Verified sources

- Berkeley Function-Calling Leaderboard (BFCL): https://gorilla.cs.berkeley.edu/leaderboard.html
- tau-bench (Sierra Research): https://github.com/sierra-research/tau-bench
- SWE-bench Verified: https://www.swebench.com/verified.html
- RouteLLM (cascade routing, arXiv:2406.18665): https://arxiv.org/abs/2406.18665

---

## Related resources

- [Evaluating AI Agents: Benchmarks and Methods](https://changegamer.ai/resources/evaluating-ai-agents.md): Why agent eval differs from single-turn LLM eval, a verified benchmark reference table (SWE-bench, GAIA, BFCL, tau-bench, WebArena, AgentBench, MLE-bench, OSWorld), and practical evaluation methods for agent builders.
- [Agent Cost and Latency Optimization](https://changegamer.ai/resources/agent-cost-latency-optimization.md): Practitioner reference for reducing the cost and latency of production AI agents: the compounding model, token-level levers (caching, pruning), request-level levers (Batch API, parallelism), model-level levers (routing, reasoning-effort controls), and architecture-level levers (step reduction, semantic caching, code offloading).
- [Application-Level Response Caching for AI Agents](https://changegamer.ai/resources/agent-response-caching.md): How to implement exact-match and semantic caching in your agent application to eliminate redundant LLM calls, with threshold guidance, invalidation strategies, and a decision matrix for when semantic caching is unsafe.
- [Prompt Caching for AI Agents](https://changegamer.ai/resources/prompt-caching-for-agents.md): Cross-provider prompt caching reference: how to activate it, minimum token thresholds, TTLs, read-vs-write pricing, and when it pays off for agentic workloads.

---

Index of all resources: https://changegamer.ai/llms.txt · Full corpus: https://changegamer.ai/llms-full.txt · Corpus data (NDJSON): https://changegamer.ai/api/corpus.jsonl · Offers: https://changegamer.ai/api/pricing.json