{
  "slug": "reliable-tool-calling",
  "title": "Reliable Tool Calling and Structured Outputs",
  "description": "How providers guarantee schema-valid tool calls and structured output — mechanisms, failure modes, and mitigations — for production agent builders.",
  "category": "Guide",
  "tags": [
    "tool-calling",
    "structured-outputs",
    "json-mode",
    "constrained-decoding",
    "agents",
    "reliability"
  ],
  "updated": "2026-08-09",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/reliable-tool-calling",
  "markdown": "https://changegamer.ai/resources/reliable-tool-calling.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "The two primitives",
      "anchor": "the-two-primitives"
    },
    {
      "depth": 2,
      "text": "Constrained decoding: how providers guarantee schema-valid output",
      "anchor": "constrained-decoding-how-providers-guarantee-schema-valid-output"
    },
    {
      "depth": 2,
      "text": "Reliability failure modes and mitigations",
      "anchor": "reliability-failure-modes-and-mitigations"
    },
    {
      "depth": 2,
      "text": "Evaluation",
      "anchor": "evaluation"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "structured-outputs-and-json-mode",
      "title": "Structured Outputs and JSON Mode: Provider Reference",
      "description": "How to force schema-valid JSON from OpenAI, Anthropic, and Gemini — parameter names, strict-mode requirements, schema-subset limits, and self-hosted constrained decoding.",
      "url": "https://changegamer.ai/resources/structured-outputs-and-json-mode"
    },
    {
      "slug": "durable-execution-for-agents",
      "title": "Durable Execution for Long-Running Agents",
      "description": "Vendor-neutral reference on durable execution: event logs, replay determinism, idempotency, retries, and human-in-the-loop pause/resume — plus a cross-vendor survey and tradeoffs guide for Temporal, Restate, DBOS, Inngest, Step Functions, Azure Durable Functions, Cloudflare Workflows, GCP Workflows, LangGraph, and OpenAI Agents SDK.",
      "url": "https://changegamer.ai/resources/durable-execution-for-agents"
    },
    {
      "slug": "handling-rate-limits-and-retries",
      "title": "Handling LLM Rate Limits (HTTP 429) and Retries for Agents",
      "description": "A practical reference for agent builders: what a 429 means, how to read provider rate-limit headers, exponential backoff with jitter, client-side throttling, and when to use a batch API.",
      "url": "https://changegamer.ai/resources/handling-rate-limits-and-retries"
    },
    {
      "slug": "choosing-an-llm-for-agents",
      "title": "How to Choose an LLM for Agentic Tasks",
      "description": "A criteria-based decision framework for selecting an LLM for agent use: tool-calling reliability, long-context behavior, structured output, cost per task, latency, and a step-by-step selection procedure.",
      "url": "https://changegamer.ai/resources/choosing-an-llm-for-agents"
    }
  ],
  "furtherReading": [
    {
      "slug": "mcp-server-failure-modes",
      "title": "Common MCP Server Failure Modes and How to Fix Them",
      "description": "A runtime playbook for the two MCP server failure modes with no dedicated deep-dive elsewhere: unrecoverable state after a mid-call crash, and malformed or hallucinated tool calls that reach the handler despite upstream validation.",
      "url": "https://changegamer.ai/articles/mcp-server-failure-modes"
    },
    {
      "slug": "mcp-tool-description-injection",
      "title": "Defending MCP Clients Against Tool Description and Output Injection",
      "description": "Two distinct MCP injection surfaces — a tool description at connect-time and a tool's return value at call-time — and the client-side architectural patterns (Dual LLM, Action-Selector, Context-Minimization) that contain each one.",
      "url": "https://changegamer.ai/articles/mcp-tool-description-injection"
    }
  ],
  "body": "Tool-call and JSON reliability is the single most important property for production agents. A model that hallucinates a tool name, drops a required argument, or emits malformed JSON turns every downstream step into an error-handling problem. This guide covers the two primitives, how each major provider enforces them, common failure modes, and concrete mitigations.\n\n## Key facts\n\n- A tool call is an in-flight instruction for the calling application to execute, whereas JSON mode / structured outputs shape only the model's concluding response — since the two mechanisms operate independently, they can be required together or separately.\n- Provider-side guarantees vary: OpenAI relies on a strict-mode schema flag, Anthropic combines per-tool strict schemas with a separate Structured Outputs config, Gemini forces compliance through ANY-mode function calls or a response schema, and self-hosted models push constraints down to token sampling via grammars such as GBNF, XGrammar, or Outlines.\n- Typical breakdowns range from invented tool names and skipped or extraneous arguments to unparsable JSON, mistyped fields, needless or missing invocations, out-of-order parallel calls, and templates that don't match how the model was trained — each pairs with its own fix.\n- Broadly applicable fixes include swapping free text for enumerated or constant values, trimming schemas down to few required fields, layering a validate-and-resubmit step on top of raw output, showing correct examples up front, and dialing down sampling temperature when hardware-level constraints aren't available.\n- BFCL, built by the Gorilla team at Berkeley, is treated as the reference benchmark; it grades both single and simultaneous calls through syntax-tree comparison, and its fourth iteration now folds in multi-step agent-style tasks.\n\n## The two primitives\n\n**Function / tool calling** — the model's intermediate output: it emits a structured call to a named tool (function name + arguments) instead of, or in addition to, a text reply. The calling application executes the tool and returns results for the model to incorporate. Tool calls appear mid-conversation; they are not the final answer.\n\n**Structured outputs / JSON mode** — the model's *final* answer is constrained to a schema. No tool is called; the response itself must be valid JSON (or match a stricter JSON Schema). Use this when you need the model's conclusion in a machine-parseable form, not when you need it to invoke external functions.\n\nThe two are orthogonal: you can require tool calls without constraining the surrounding text, constrain the final answer without any tools, or combine both.\n\n## Constrained decoding: how providers guarantee schema-valid output\n\n**OpenAI — Structured Outputs (strict mode)**\nSet `strict: true` on a function definition or response format. OpenAI's constrained decoding then guarantees the output matches the supplied JSON Schema exactly. Requirements: every object must set `additionalProperties: false`; every property must appear in the `required` array (mark optional fields with a union type that includes `null`). Without `strict: true`, the model uses best-effort JSON mode, which does not guarantee schema compliance. Source: platform.openai.com/docs/guides/function-calling (Structured Outputs section).\n\n**Anthropic — tool use + `tool_choice` + Structured Outputs**\nPass tool schemas via the `tools` array. `tool_choice` has four options: `{\"type\": \"auto\"}` (default — Claude decides), `{\"type\": \"any\"}` (must use at least one of the provided tools), `{\"type\": \"tool\", \"name\": \"<name>\"}` (must call that specific tool), and `{\"type\": \"none\"}` (cannot use tools). Any of them can additionally carry `\"disable_parallel_tool_use\": true` to cap Claude at one tool call per response. For guaranteed schema-valid tool *inputs*, set `strict: true` as a top-level field on the tool definition (not on `tool_choice`); the schema must set `additionalProperties: false` and list `required` fields. For constrained final-answer JSON, Anthropic now ships **Structured Outputs**: `output_config: {\"format\": {\"type\": \"json_schema\", \"schema\": ...}}` on the Messages API (the older top-level `output_format` parameter is deprecated), with SDK `messages.parse()` helpers that validate against a Pydantic/Zod schema. Supported on current models (and recent legacy Opus releases); schema limitations apply — no recursive schemas, no numeric/string constraints (`minimum`, `maxLength`, …), and objects must set `additionalProperties: false`. This supersedes the earlier pattern of faking JSON mode through a named tool. Sources: platform.claude.com/docs/en/agents-and-tools/tool-use/overview; platform.claude.com/docs/en/build-with-claude/structured-outputs.\n\n**Google Gemini — function calling + `tool_config` + `responseSchema`**\nFunction calling uses `tool_config.function_calling_config.mode`: `AUTO` (model decides), `ANY` (must call a function — guarantees schema-typed output), or `NONE` (no tool calls). `ANY` mode with a function declaration gives schema adherence comparable to strict mode. For final-answer structured output, set `response_mime_type: \"application/json\"` and `response_schema` in the generation config. Note: `response_schema` and tool calling have historically been mutually exclusive in any mode — declaring both errors. Gemini 3 models relax this: Google's docs now describe a `VALIDATED` mode that combines function calling with a response schema for consistently formatted non-call replies (WebSearch-corroborated only this cycle — ai.google.dev was unreachable through this proxy, so treat the `VALIDATED` detail as directionally correct pending direct confirmation). Source: ai.google.dev/gemini-api/docs/function-calling; ai.google.dev/gemini-api/docs/structured-output.\n\n**Open models — grammar-based generation**\nLocal inference runtimes enforce schemas at the token-sampling layer. Two mainstream approaches:\n\n- *llama.cpp GBNF* — GGML BNF (GBNF), an extension of Backus-Naur Form, defines grammars that constrain token selection. Pass a grammar string or a JSON Schema (auto-converted to GBNF) at inference time. Note: grammar and function-calling cannot be used simultaneously in llama.cpp; function calling uses its own internal grammar. Source: github.com/ggml-org/llama.cpp/blob/master/grammars/README.md.\n\n- *XGrammar* — the default structured-generation backend for vLLM, SGLang, TensorRT-LLM, and MLC-LLM as of 2025. Compiles JSON Schema / EBNF to a pushdown automaton; applies bitwise token masking at under 40 µs overhead per token. Source: github.com/mlc-ai/xgrammar.\n\n- *Outlines (dottxt-ai)* — Python library that compiles JSON Schema or regex constraints to finite-state machines and masks invalid tokens during sampling. Works with Transformers, vLLM, Ollama, and others. Source: github.com/dottxt-ai/outlines.\n\n## Reliability failure modes and mitigations\n\n| Failure mode | Mitigation |\n|---|---|\n| **Hallucinated tool name** — model calls a tool not in the declared set | Validate the returned tool name against your schema before executing; reject unknown names |\n| **Missing required arguments** — model omits a field the schema marks required | Use strict mode / `required` + `additionalProperties: false`; validation library catches missing fields pre-execution |\n| **Extra / unexpected arguments** — model adds fields not in schema | `additionalProperties: false` (OpenAI strict) or schema validation; strip unknown keys defensively |\n| **Malformed JSON** — output is not parseable | Enable provider-level strict mode or constrained decoding; wrap parse in try/catch and retry with an error message |\n| **Wrong types** — string where int expected, etc. | Declare enum or `const` values where possible; use schema validation (e.g. Pydantic, Zod) before consuming arguments |\n| **Over-calling** — model calls tools unnecessarily | Use `tool_choice: \"auto\"` and a minimal toolset; evaluate on your task distribution, not just benchmark scores |\n| **Under-calling** — model answers in text when a tool call was required | Force tool use via `tool_choice: \"any\"` (Anthropic), `mode: \"ANY\"` (Gemini), or remove text-only response option entirely |\n| **Parallel tool calls in wrong order** — parallel calls with dependencies | Declare dependencies explicitly; set `disable_parallel_tool_use: true` (Anthropic) or sequential `tool_choice` forcing when order matters |\n| **Chat-template mismatch** — open model's tool schema injected with wrong template | Always match the inference framework's chat template exactly to the model's training template; mismatches silently degrade reliability |\n\n**Cross-cutting mitigations:**\n\n- Prefer enums and `const` values over free-text fields wherever the value space is bounded.\n- Keep schemas shallow and required fields minimal — every optional field is a reliability risk.\n- Add a validation + retry/repair loop: parse and validate the model's output; on failure, send the validation error back as a user message and request a corrected call.\n- Use few-shot examples of correct tool calls in the system prompt.\n- Lower temperature (toward 0) improves schema adherence on models without constrained decoding.\n\n## Evaluation\n\nThe standard benchmark for tool-calling reliability is the **Berkeley Function Calling Leaderboard (BFCL)**, maintained by Gorilla LLM at UC Berkeley. BFCL evaluates serial and parallel function calls across multiple languages using Abstract Syntax Tree (AST) scoring. BFCL V4 (current) extends evaluation to multi-turn and holistic agentic scenarios. Live leaderboard: gorilla.cs.berkeley.edu/leaderboard.html.\n\nFor open-weight model tool-calling scores and license terms, see /resources/open-weight-models-for-agents. For validating tool outputs as a security control, see /resources/agentic-security-checklist.\n\n## Verified sources\n\n- OpenAI function calling (includes Structured Outputs strict mode): https://platform.openai.com/docs/guides/function-calling\n- OpenAI Structured Outputs guide: https://developers.openai.com/api/docs/guides/structured-outputs\n- Anthropic tool use — implement tool use: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/implement-tool-use\n- Anthropic Structured Outputs: https://platform.claude.com/docs/en/build-with-claude/structured-outputs\n- Anthropic tool choice overview: https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview\n- Google Gemini function calling: https://ai.google.dev/gemini-api/docs/function-calling\n- Google Gemini structured output: https://ai.google.dev/gemini-api/docs/structured-output\n- llama.cpp GBNF grammar README: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md\n- XGrammar (mlc-ai, default backend for vLLM/SGLang/TensorRT-LLM): https://github.com/mlc-ai/xgrammar\n- Outlines (dottxt-ai, constrained decoding library): https://github.com/dottxt-ai/outlines\n- Berkeley Function Calling Leaderboard (BFCL V4): https://gorilla.cs.berkeley.edu/leaderboard.html",
  "sources": [
    "https://platform.openai.com/docs/guides/function-calling",
    "https://developers.openai.com/api/docs/guides/structured-outputs",
    "https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/implement-tool-use",
    "https://platform.claude.com/docs/en/build-with-claude/structured-outputs",
    "https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview",
    "https://ai.google.dev/gemini-api/docs/function-calling",
    "https://ai.google.dev/gemini-api/docs/structured-output",
    "https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md",
    "https://github.com/mlc-ai/xgrammar",
    "https://github.com/dottxt-ai/outlines",
    "https://gorilla.cs.berkeley.edu/leaderboard.html"
  ]
}