← All guides · Agent reliability in production

How to Make AI Agent Tool Calling Reliable

Part 1 of Agent reliability in production · 1,791 words · ~8 min read · published 2026-08-26 · updated 2026-08-26 · Markdown variant

An operator playbook for tool-calling contracts: per-provider strict-mode config (OpenAI, Anthropic, Gemini, self-hosted grammars), the full failure-mode-to-fix table, what BFCL actually measures, and how to stop parallel tool calls from breaking a dependency chain.

Part of the Agent Guardrails and the AI Agent Reliability Playbook guide.

Tool-calling reliability is one discipline inside the twelve-part stack the agent reliability in production pillar surveys — this article deep-dives that one discipline: what a tool-calling contract enforces, how each provider implements it, the failure modes a strict contract still doesn't eliminate, and how call ordering breaks under naive parallel execution.

What makes a tool call reliable enough to trust in production?

A tool call is reliable enough to trust in production when the calling application never has to guess whether the model's output is schema-valid — when a decoding-level guarantee rules out a malformed shape outright, rather than the application hoping a well-written prompt was enough. That is the core distinction between a "contract" and a suggestion: a prompt asking the model to "always return valid JSON" is advice it can still ignore under a long context or ambiguous input; a schema flagged strict: true with additionalProperties: false is a constraint the decoder cannot violate, because the sampler itself excludes any token that would break the declared shape. As of August 2026 all three major hosted providers — OpenAI, Anthropic, Google Gemini — ship some version of this guarantee, and every serious self-hosted inference stack enforces the equivalent at the sampling layer through a grammar. None of them eliminates every failure mode below; a valid shape is not the same as a correct value, and that gap is where the rest of this playbook lives.

How does each provider enforce a tool-calling contract?

Each of the three major hosted providers enforces schema validity through a different mechanism, and porting a tool contract between them means re-implementing the guarantee, not just re-declaring the same schema.

OpenAI: strict mode

Set strict: true on a function definition. OpenAI's constrained decoding then guarantees the output matches the schema exactly, provided every object sets additionalProperties: false and every property appears in required — model an optional field as a nullable union, not an omission from required:

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "location": { "type": "string" },
        "unit": { "type": ["string", "null"], "enum": ["celsius", "fahrenheit"] }
      },
      "required": ["location", "unit"],
      "additionalProperties": false
    }
  }
}

Without strict: true, the call falls back to best-effort generation with no compliance guarantee — the flag is the entire mechanism, not a tuning knob.

Anthropic: strict tool definitions plus separate output control

Anthropic's tool schemas carry their own strict: true field on the tool definition, constraining that tool's arguments specifically — it says nothing about the model's own final reply, which a separate output_config mechanism governs instead. tool_choice independently controls whether a tool is used at all: auto lets the model decide, any forces at least one call, tool with a name forces that specific tool, and none blocks tool use.

{
  "name": "get_weather",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "location": { "type": "string" },
      "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
    },
    "required": ["location", "unit"],
    "additionalProperties": false
  }
}

Gemini: ANY-mode function calls

Gemini enforces schema-typed tool output by setting tool_config.function_calling_config.mode to ANY, forcing the model to call a declared function rather than answer in free text, with adherence Google's own documentation describes as comparable to the strict-mode guarantees above:

{
  "tool_config": {
    "function_calling_config": {
      "mode": "ANY",
      "allowed_function_names": ["get_weather"]
    }
  }
}

Historically, Gemini's function calling and its responseSchema for a final answer have been mutually exclusive within a single call — declaring both errors. Google's docs reportedly describe a Gemini 3 VALIDATED mode combining the two, a detail the reference corpus carries as WebSearch-corroborated only this cycle and not independently confirmed by a direct fetch of Google's own pages — treat it as directionally likely, not settled.

Self-hosted models: grammar-constrained sampling

Self-hosted models have no provider API to flag strict: true against, so the constraint moves into the inference runtime itself. llama.cpp compiles a GBNF (GGML BNF) grammar — or an auto-converted JSON Schema — into rules that restrict which tokens the sampler can choose:

# Illustrative — the shape of a GBNF grammar constraining a tool call's
# output to a fixed JSON structure, not tied to a specific llama.cpp
# version. grammar and function-calling cannot be used simultaneously in
# llama.cpp; function calling uses its own internal grammar.
root    ::= "{" ws "\"tool\"" ws ":" ws "\"get_weather\"" ws "," ws "\"location\"" ws ":" ws string "}"
string  ::= "\"" [^"]* "\""
ws      ::= [ \t\n]*

vLLM, SGLang, TensorRT-LLM, and MLC-LLM instead default to XGrammar as of 2025, exposed through a guided_json parameter compiling a JSON Schema to a pushdown automaton at well under 40 microseconds of token-masking overhead; Outlines is a third option, compiling a schema or regex to a finite-state machine on the same runtimes. All three guarantee the same thing strict mode guarantees — a valid shape — under the operator's own control rather than a provider flag.

The tool-calling failure-mode-to-fix table

Real tool calls still fail in a small, well-documented set of shapes even under full strict-mode or grammar enforcement, and each one needs its own fix rather than one universal validator.

Two mitigations cut across nearly every row: trim each schema down to the smallest possible set of required fields, since a field left optional is a slot the model is free to skip, garble, or invent a value for; and build a resubmission loop around raw output — parse it, validate it, and on failure hand the exact validator error back to the model in a follow-up turn instead of letting a bad call crash the calling application outright. A related but distinct discipline lives one layer downstream: common MCP server failure modes covers what an MCP handler does when a malformed or hallucinated call reaches it despite this contract, and how it recovers from a crash mid-call — this article stops at the contract that produces the call.

What does the Berkeley Function Calling Leaderboard actually measure?

BFCL (the Berkeley Function Calling Leaderboard) measures tool-call correctness by comparing a model's emitted call against a reference answer through abstract-syntax-tree comparison, across both serial and parallel call scenarios — the Gorilla team at UC Berkeley builds and runs it. AST comparison checks the structure of the call — the tool name and argument tree — against a known-correct reference rather than scoring free-text similarity, which makes it harder to game with verbose or plausible-sounding output than an LLM-as-judge approach. Its fourth iteration, current as of August 2026, extends the leaderboard beyond isolated single calls to fold in multi-step, agentic tasks — a model can pass a single-call AST check perfectly and still fail a multi-step BFCL task if it gets the order of dependent calls wrong, a distinct failure from getting any individual call's shape wrong.

Why do parallel tool calls break dependency chains, and how do you force ordering?

Parallel tool calls break a dependency chain when one call's correct arguments depend on a value only a prior call can return, and the model issues both calls in the same turn instead of waiting for the first result. Consider an agent refunding a customer: it must first call lookup_order to get an order ID, then call issue_refund with that ID. If the model treats both as independent and parallelizable — a shape it has no inherent reason to avoid unless told to — issue_refund fires with a fabricated or empty order ID before lookup_order's result ever returns, because nothing in a naive parallel tool-call turn guarantees one call's output reaches another call issued in the same turn.

The fix makes the dependency explicit rather than implicit in the prompt, using ordering controls each provider already exposes:

• Anthropic — set disable_parallel_tool_use: true on tool_choice to cap the model at one call per turn, guaranteeing the second call in a chain only fires after the first call's result is back in context.
• Forced sequential tool_choice — on any provider that supports naming a specific tool, force lookup_order first and only expose issue_refund once its result is available, rather than declaring both tools simultaneously and hoping the model sequences them correctly.

Neither fix requires the model to reason about dependency order on its own — both remove the opportunity for a parallel call to fire before its precondition exists, which is more reliable than trusting a prompt instruction like "call these in order" to hold under every input the agent will see in production.

Where this leaves you

Provider-side strict mode and grammar-based decoding solve the shape problem — a call that cannot be malformed — but not the value or ordering problem, which is why the failure-mode table and the parallel-call fix above both still matter with strict mode fully enabled. Validate tool names and argument values defensively regardless of which provider's guarantee you rely on, force sequential execution wherever one call's arguments depend on another call's result, and treat BFCL as a proxy for cross-model comparison rather than a guarantee about your own task distribution's failure rate. The full mechanism and mitigation reference this article draws from is reliable tool calling and structured outputs; for the other eleven reliability disciplines this fits inside, see the agent reliability in production pillar.

#agents #reliability #tool-calling #constrained-decoding #bfcl #production