# Structured Outputs vs Tool Calling: When to Use Each

> A decision framework for structured outputs versus tool calling in AI agents, with runnable JSON Schema examples for OpenAI, Anthropic, Gemini, vLLM/SGLang, and llama.cpp, plus mitigation code for truncation, refusal, and grammar-compilation latency.

Guide: Agent reliability in production — part 2
Published: 2026-08-26 · Updated: 2026-08-26 · 1883 words · ~2504 tokens (estimate)
Canonical: https://changegamer.ai/articles/structured-outputs-vs-tool-calling
JSON: https://changegamer.ai/api/articles/structured-outputs-vs-tool-calling.json
Pillar: https://changegamer.ai/articles/agent-reliability-in-production.md

## In short

- Structured outputs constrain a model's own final reply into a schema-valid object with no function invoked, while tool calling produces an intermediate instruction for the calling application to execute — a single agent turn can need either mechanism alone or both at once, since neither one substitutes for the other.
- OpenAI's json_schema strict mode disallows a root-level anyOf and any default value, Anthropic's output_config.format supports anyOf and $ref but not recursion, and Gemini splits into an OpenAPI-subset responseSchema for 1.5/2.0 versus a full-JSON-Schema responseJsonSchema for 2.5 and later — porting one schema across all three requires checking each provider's subset rather than assuming portability.
- A schema-valid guarantee under strict mode is bypassed by two conditions that a 200 status code does not rule out: a stop_reason of max_tokens truncating the object mid-generation, or a stop_reason of refusal overriding the schema entirely with a safety response.
- Self-hosted inference stacks enforce structured output at the token-sampling layer rather than through a provider flag — vLLM and SGLang expose this as a guided_json parameter compiled by XGrammar, and llama.cpp compiles a GBNF grammar directly.
- The first request against a new or changed JSON Schema on Anthropic and self-hosted grammar backends is measurably slower than subsequent requests because the schema has to compile into a grammar before generation starts, which a warm-up call issued at deploy time avoids exposing to a real user.

---

Structured outputs and tool calling sit side by side in one summary table inside the [agent reliability in production](/articles/agent-reliability-in-production) pillar. This article expands only the structured-outputs half into a full decision guide: a framework for picking a schema-constrained reply over a function call, the per-provider schema subsets shown in runnable code, and both failure modes that survive even under a strict-mode guarantee. It does not re-derive tool-calling's own failure-mode table, BFCL scoring, or parallel-call ordering — see [tool-calling contracts for AI agents](/articles/tool-calling-contracts-for-ai-agents) for that half of the picture.

## When should you reach for structured outputs instead of a tool call?

Reach for structured outputs when the task ends with the model's own answer in machine-parseable form, and reach for tool calling when the task requires your application to execute an external action. The distinction is about what happens after the model responds, not about how complex the output schema is. Sentiment-tag a support transcript into `{"sentiment": "frustrated", "priority": 2}` and nothing outside the model has to run — the object itself is the deliverable, so structured outputs are the right mechanism. Look up a customer's subscription tier and cancel a pending renewal, and the model's output has to name a function and its arguments for your application code to execute — that is a tool call, not a final answer.

Neither mechanism substitutes for the other: a single agent turn can need one alone, both in sequence, or both inside the same call depending on what the provider allows. Three task shapes make this concrete:

- **Extraction and classification** — pulling structured fields out of unstructured text, or assigning a label with a confidence score. No external system needs to be touched; structured outputs alone are correct here.
- **Multi-step agentic action** — looking up a record, then acting on it. Tool calling drives the lookup and the action; structured outputs have no role unless the very last step is also a report back to the user in a fixed shape.
- **Tool call followed by a schema-valid summary** — an agent that queries an API via tool calling and then has to hand the orchestrating system a schema-valid result object combines both: the tool call for the action, structured outputs for the final reply.

Whether a given provider lets you request both mechanisms in the same call, rather than as two sequential requests, is a compatibility detail that varies by provider — covered in the FAQ below — and is worth checking before you design a contract that assumes one call does both.

## What schema subset does each provider actually support?

Each provider supports a different subset of JSON Schema under its strict-mode guarantee, and the differences are large enough that a schema portable across all of them has to be written to the narrowest common subset, not the richest one any single provider allows. As of August 2026, five schema-subset comparisons matter in practice: OpenAI, Anthropic, Gemini's two schema modes, and the two dominant self-hosted patterns.

### OpenAI: `response_format` with `json_schema` and `strict: true`

OpenAI's strict mode enforces two structural rules on every object node: `additionalProperties` must be `false`, and every declared property must be listed as `required` — model an optional field as a nullable type union rather than leaving it out of `required`. A root-level `anyOf` is rejected, and including a `default` value anywhere in the schema makes the call fail outright rather than being silently dropped.

```json
{
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "ticket_classification",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "category": { "type": "string", "enum": ["billing", "technical", "account"] },
          "confidence": { "type": "number" },
          "escalate": { "type": ["boolean", "null"] }
        },
        "required": ["category", "confidence", "escalate"],
        "additionalProperties": false
      }
    }
  }
}
```

Supported on gpt-4o (2024-08-06 and later), GPT-4.1, GPT-5, and the o-series; older snapshots fall back to the weaker `json_object` mode described below. `response_format` and `tools` are mutually exclusive within a single call.

### Anthropic: `output_config.format`

Anthropic's native structured outputs shipped in late 2025 and are GA on current models as of August 2026, with no beta header required. The canonical call-time parameter is `output_config.format` set on `messages.create` itself — a separate top-level `output_format` create-parameter exists in some SDK surfaces but is deprecated, so new integrations should target `output_config` directly rather than copying an older example that uses the deprecated name.

```json
{
  "output_config": {
    "format": {
      "type": "json_schema",
      "schema": {
        "type": "object",
        "properties": {
          "category": { "type": "string" },
          "confidence": { "type": "number" },
          "notes": {
            "anyOf": [
              { "type": "string" },
              { "type": "null" }
            ]
          }
        },
        "required": ["category", "confidence"]
      }
    }
  }
}
```

Unlike OpenAI, Anthropic's schema support includes `anyOf` and `$ref`/`$defs`, but not recursive schemas, and per-request limits apply to tool count and optional/union parameter counts — check current docs for exact caps rather than assuming last year's limits still hold. This is separate from `strict: true` on an individual tool definition, which constrains that tool's *arguments*, not the model's final reply — the two are configured independently even though both live under the Anthropic API.

### Gemini: `responseSchema` versus `responseJsonSchema` — two different modes, not one

Gemini splits schema support across model generations into two genuinely different parameters, and treating them as interchangeable is the most common integration mistake. `responseSchema`, used with `responseMimeType: "application/json"` on Gemini 1.5 and 2.0, accepts only an OpenAPI-based subset of JSON Schema — no `anyOf`, no `$ref`:

```json
{
  "generationConfig": {
    "responseMimeType": "application/json",
    "responseSchema": {
      "type": "object",
      "properties": {
        "category": { "type": "string" },
        "confidence": { "type": "number" }
      },
      "required": ["category", "confidence"]
    }
  }
}
```

Gemini 2.5 and later additionally accept `responseJsonSchema`, which supports the fuller JSON Schema feature set — `anyOf`, `$ref`, and limited recursion — and accepts a schema derived directly from a Pydantic or Zod model:

```json
{
  "generationConfig": {
    "responseMimeType": "application/json",
    "responseJsonSchema": {
      "type": "object",
      "properties": {
        "category": { "type": "string" },
        "notes": { "anyOf": [{ "type": "string" }, { "type": "null" }] }
      },
      "required": ["category"]
    }
  }
}
```

A schema written for `responseJsonSchema`'s richer feature set is not guaranteed to work unmodified if sent as `responseSchema` against an older model — check which parameter a given model generation actually accepts before assuming a schema is portable across your own Gemini model pins. Some 2.5 snapshots have also been reported to fail structured output when tool-call messages are present in history, and an overly complex schema can trigger a 400 — shortening property names, reducing enum sizes, or flattening nested arrays are the standard workarounds.

### Self-hosted: `guided_json` (vLLM, SGLang) and `grammar` (llama.cpp)

There is no hosted-provider flag to set for a self-hosted deployment, so the schema constraint has to be enforced by the serving stack itself, at the point where tokens are sampled. vLLM and SGLang expose a `guided_json` parameter that takes a JSON Schema directly; both runtimes route it through the XGrammar library by default as of 2025, which compiles the schema into a pushdown automaton with low per-token overhead:

```python
response = client.chat.completions.create(
    model="your-self-hosted-model",
    messages=[{"role": "user", "content": "Classify this support ticket."}],
    extra_body={
        "guided_json": {
            "type": "object",
            "properties": {
                "category": {"type": "string"},
                "confidence": {"type": "number"}
            },
            "required": ["category", "confidence"]
        }
    },
)
```

llama.cpp instead compiles a GBNF (GGML Backus-Naur Form) grammar, either hand-written or auto-converted from a JSON Schema, into the same kind of token-level constraint via its own `grammar` parameter — the parameter name and grammar format are specific to llama.cpp and not interchangeable with vLLM's `guided_json`. Outlines is a third self-hosted option available on the same runtimes, using finite-state-machine masking, though it does not support recursive schemas — XGrammar or llguidance are the alternatives when recursion is required. The full per-provider table this section expands is in [structured outputs and JSON mode](/resources/structured-outputs-and-json-mode); the equivalent grammar-based mechanism as it applies to tool-call arguments specifically, rather than a final-answer schema, is covered in [tool-calling contracts for AI agents](/articles/tool-calling-contracts-for-ai-agents).

## Why does a strict schema guarantee still fail in production?

A strict schema guarantee still fails in production because two conditions bypass constrained decoding entirely rather than merely stressing it: the response can be truncated before the object closes, or the model can refuse instead of generating at all. Neither condition is caught by checking the HTTP status code, because both return a normal 200 response — the schema violation is only visible in the `stop_reason` field, which is why a 200 status must never be treated as proof of a valid object.

### Checking `stop_reason` before you parse anything

A `stop_reason` of `"max_tokens"` means generation was cut off mid-object before the schema could close, and a `stop_reason` of `"refusal"` means a safety response overrode the schema entirely — both need to be checked as first-class branches before any parsing is attempted:

```python
def get_structured_response(client, request):
    response = client.messages.create(**request)

    if response.stop_reason == "max_tokens":
        raise TruncatedOutputError(
            "Response cut off before schema closed; raise max_tokens or simplify the schema"
        )
    if response.stop_reason == "refusal":
        raise SchemaRefusalError(
            "Model refused instead of generating a schema-valid object"
        )

    # Only now is a 200-status response actually safe to parse.
    return json.loads(response.content[0].text)
```

Treating these as parse-time exceptions rather than upstream validation failures is the difference between a crash your monitoring never explains and a distinguishable, loggable failure mode your retry-and-repair loop can act on directly.

### First-request grammar-compilation latency

As of August 2026, the first request against a new or changed JSON Schema is measurably slower than subsequent requests on Anthropic and on every self-hosted grammar backend, because the schema has to compile into a grammar before generation can start, and a schema change invalidates any cached compilation. A production deploy that only sees this delay on a real user's first request has shipped the mitigation in the wrong place — a warm-up call at deploy time, before traffic arrives, moves the cost off the request path entirely:

```python
def warm_up_schema_cache(client, schema, model):
    """Call once per deploy, per distinct schema, before serving real traffic.
    Absorbs first-request grammar-compilation latency outside the request path."""
    client.messages.create(
        model=model,
        max_tokens=16,
        messages=[{"role": "user", "content": "warm-up: reply with a minimal valid object"}],
        output_config={"format": {"type": "json_schema", "schema": schema}},
    )
```

The other side of this mitigation is discipline, not code: avoid frequent schema churn in the same way you'd avoid frequent cache-key churn, since each distinct schema version pays this compilation cost again on its own first request regardless of how many times an earlier version of the schema already warmed up.

## Where this leaves you

The task shape decides the mechanism: pick structured outputs when the last step is the model's own answer in a fixed shape, and pick tool calling when the last step is an action your application has to execute. Check each provider's actual schema subset before assuming a contract written for one provider ports cleanly to another, since OpenAI's, Anthropic's, and Gemini's two schema modes each draw the line differently. Whichever mechanism you use, treat `stop_reason` as a required check before parsing and warm up a new schema's grammar before real traffic hits it — a strict-mode guarantee only holds for the requests that actually reach a complete, non-refused generation. For the twelve-discipline reliability stack this decision sits inside, see the [agent reliability in production](/articles/agent-reliability-in-production) pillar; for the orthogonal question of making the tool-calling side of an agent's output reliable, see [tool-calling contracts for AI agents](/articles/tool-calling-contracts-for-ai-agents).

## Frequently asked questions

### Can you use structured outputs and tool calling in the same request?

Yes on some providers and no on others as of August 2026: OpenAI's response_format and tools parameters are mutually exclusive in a single call, so a request needing both a tool invocation and a schema-valid final reply has to run them as two separate calls, while Anthropic's output_config for the final reply and strict: true on a tool definition are independent mechanisms that can be configured together in one request.

### Is JSON mode the same as structured outputs?

No, JSON mode (OpenAI's json_object) only asks the model to produce syntactically valid JSON with no schema enforced, so it can still omit required fields, use the wrong type, or add unexpected keys, whereas structured outputs (json_schema with strict: true) constrain the decoder so it physically cannot emit a token that violates the declared schema — OpenAI now treats json_object as a legacy mode and recommends json_schema wherever a model supports it.

### Why did my structured output request return invalid JSON even with strict mode on?

A structured output request returns invalid or incomplete JSON despite strict mode when the response was cut off by a stop_reason of max_tokens before the object closed, or when the model issued a safety refusal that carries a stop_reason of refusal instead of a completed generation — both bypass the schema guarantee, so checking the stop reason before attempting to parse the response is a required step, not an optional one.

### Does Gemini use the same schema format for every model version?

No, Gemini 1.5 and 2.0 accept only responseSchema, an OpenAPI-based subset of JSON Schema with no anyOf or $ref support, while Gemini 2.5 and later also accept responseJsonSchema, which supports the fuller JSON Schema feature set including anyOf, $ref, and limited recursion — a schema written for one Gemini schema mode is not guaranteed to work unmodified against the other.


---

## The rest of this guide

- [Agent Guardrails and the AI Agent Reliability Playbook](https://changegamer.ai/articles/agent-reliability-in-production.md): Agent guardrails plus the eleven other disciplines that make an AI agent reliable in production: tool calling, retries, durable execution and rollout.
- [How to Make AI Agent Tool Calling Reliable](https://changegamer.ai/articles/tool-calling-contracts-for-ai-agents.md): An operator playbook for tool-calling contracts: per-provider strict-mode config (OpenAI, Anthropic, Gemini, self-hosted grammars), the full failure-mode-to-fix table, what BFCL actually measures, and how to stop parallel tool calls from breaking a dependency chain.
- [How to Make AI Agent Retries Idempotent](https://changegamer.ai/articles/retries-and-idempotency-for-ai-agents.md): A deep-dive on retrying agent tool calls safely: the transient-vs-terminal decision, why an agent side effect can fire before a failure signal reaches the caller, idempotency-key mechanics (run ID + step index), the unknown-outcome edge case, and where idempotency keys do not reach.
- [When Do AI Agents Need Durable Execution?](https://changegamer.ai/articles/durable-execution-for-ai-agents.md): A deep-dive on durable execution for AI agents: the persisted event log, the replay-determinism constraint, the four architectural shapes mapped across ten engines and frameworks, and a decision framework for when a durable execution engine is worth adding at all.
- [How to Design Guardrails for AI Agent Reliability](https://changegamer.ai/articles/agent-guardrails-for-reliability.md): An operator playbook for reliability guardrails: the three checkpoints (input, output, action), layering cheap checks under slow ones with a fail-closed default, the two-of-three-properties rule for when a tool call needs human approval, and logging every verdict against the run trace ID.
- [How to Evaluate AI Agents in CI](https://changegamer.ai/articles/evaluating-ai-agents-in-ci.md): An operator playbook for gating an AI agent release in CI: why agent eval needs trajectory-level scoring across the tasks it actually runs, how public benchmarks diverge as proxies, ground-truth vs LLM-as-judge tool-call scoring, and the three-layer test pyramid that keeps CI fast and non-flaky.
- [How to Roll Out a New AI Agent Version Safely](https://changegamer.ai/articles/agent-rollout-and-rollback.md): An operator playbook for shipping a new agent version without breaking production: in-repo vs. registry prompt storage, a version-numbering comparison, the six-step promotion flow, A/B-test mechanics, the composite-version trace fields, and a rollback drill.
- [How to Build an Incident Response Runbook for AI Agent Failures](https://changegamer.ai/articles/agent-incident-response-runbooks.md): An operator playbook for the moment an AI agent fails in production: a triage step to classify the failure fast, trace freezing before rollback destroys the evidence, first-response depth on the four failure classes, and a blameless post-mortem that feeds back into guardrails and eval.
- [How to Set Timeouts for AI Agent Tool Calls](https://changegamer.ai/articles/timeouts-and-deadlines-for-ai-agents.md): A deep-dive on timeout and deadline design for AI agents: sizing LLM-call, tool-call, and sub-agent-hop timeouts differently, allocating a wall-clock budget across a multi-step chain, and propagating a remaining-deadline value from parent to child calls.
- [How to Design a Circuit Breaker for AI Agents](https://changegamer.ai/articles/circuit-breakers-and-degraded-mode-for-ai-agents.md): A deep-dive on the circuit breaker pattern for AI agents: the Closed/Open/Half-Open state machine with a worked open-source example, where to place a breaker in an agent's call path, and degraded-mode fallback design as its own discipline per dependency type.
- [What Should an AI Agent's Observability System Capture?](https://changegamer.ai/articles/agent-observability-for-reliability.md): An operator playbook for instrumenting an AI agent: the trace/span model behind a run, the OpenTelemetry GenAI attributes that name each field, the fields worth capturing per span, and what to redact before any of it gets logged.
- [The AI Agent Production Reliability Checklist](https://changegamer.ai/articles/agent-reliability-production-checklist.md): A go/no-go checklist that turns the agent reliability pillar's twelve-discipline closing list into checkable gates — the specific artifact, header, or trace field that proves each one holds, with a link to whichever sibling article owns its mechanics.

## Reference resources

- https://changegamer.ai/resources/structured-outputs-and-json-mode.md
- https://changegamer.ai/resources/reliable-tool-calling.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt
Licensing: https://changegamer.ai/api/pricing.json (offer catalog) · https://changegamer.ai/api/payment.json (payment methods, HTTP 402 flow) · access guide: https://changegamer.ai/resources/access-and-pricing.md