← All resources

Structured Outputs and JSON Mode: Provider Reference

Guide · updated 2026-07-02 · Markdown variant

How to force schema-valid JSON from OpenAI, Anthropic, and Gemini — parameter names, strict-mode requirements, schema-subset limits, and self-hosted constrained decoding.

Structured outputs force an LLM to return valid JSON that conforms to a specific schema, without requiring tool calls. This is distinct from tool calling: the model's final reply is the schema-constrained object, not an intermediate function invocation. See /resources/reliable-tool-calling for the tool-calling primitive and how they interact.

Key facts

• Structured outputs constrain the model's own final reply into a schema-valid object, which is a distinct mechanism from tool calling's intermediate function invocation.
• True schema-constrained decoding (json_schema / strict mode) physically prevents schema-violating tokens, whereas legacy JSON mode only asks the model for syntactically valid JSON with no schema enforced — which is why OpenAI now treats plain JSON mode as legacy.
• Each provider enforces and limits schemas differently: OpenAI's strict mode disallows default values and root-level anyOf, Anthropic's GA structured outputs support anyOf/$ref but not recursion, and Gemini splits into an OpenAPI-subset schema for older models versus a fuller JSON Schema for 2.5+.
• Self-hosted serving stacks push schema enforcement down to token sampling via grammars — vLLM and SGLang default to the XGrammar library for this, while llama.cpp relies on its own GBNF grammar format.
• Even under a strict schema guarantee, output can still come back non-compliant in practice if generation hits a token-limit truncation or a safety refusal — so the stop reason always needs checking.
• First-request latency can rise because of grammar compilation, which is mitigated by a warm-up call and avoiding frequent schema churn.
• Major SDKs from OpenAI, Anthropic, and Gemini all offer helpers that derive the schema directly from a Pydantic or Zod model and parse the result.

Which mode to use

Structured outputs / json_schema — the provider's constrained decoder physically prevents the model from emitting tokens that violate your schema. Guarantees schema compliance given no refusal and no token-limit truncation.

JSON mode / json_object — the model is merely instructed to produce syntactically valid JSON; no schema is enforced, so it may omit fields, use wrong types, or add keys. OpenAI now labels this mode legacy — prefer json_schema with strict: true.

Provider mechanisms

OpenAI

Set response_format: { type: "json_schema", json_schema: { name, schema, strict: true } }. With strict: true, constrained decoding guarantees the output matches the schema. Requirements: every object must set additionalProperties: false; every property must appear in required (make optional fields a nullable type union). Supported on gpt-4o (2024-08-06+), GPT-4.1, GPT-5, and the o-series. Older snapshots support only the weaker json_object mode.

Schema subset — not supported by strict mode: default values (the call fails); root-level anyOf (use a nullable type union instead); numeric/string constraints like minimum/pattern (accepted but not enforced by the decoder); very deep nesting / very large property counts. response_format and tools are mutually exclusive in a single call.

Anthropic (Claude)

Native structured outputs shipped in late 2025 and are now GA on current models — no beta header required. Set output_config: { format: { type: "json_schema", schema } }; the schema is compiled to a grammar and generation is constrained at decode time. Also supports strict: true on tool definitions for guaranteed tool-input validation. Notes: anyOf/$ref/$defs supported; numeric range and recursive schemas are not; per-request limits apply (caps on tool count, optional/union parameters, and grammar-compilation time — check the docs for current values); a schema change invalidates the grammar cache, adding first-request latency; stop_reason: "refusal" or "max_tokens" bypasses the guarantee, so always check stop_reason.

Google Gemini

Set responseMimeType: "application/json" together with a schema. Two schema formats: responseSchema (an OpenAPI-based subset; no anyOf/$ref; Gemini 1.5/2.0) and responseJsonSchema (full JSON Schema with anyOf/$ref; Gemini 2.5+, accepts Pydantic/Zod schemas). Known limitation: in some 2.5 snapshots structured output fails when tool-call messages are in history; overly complex schemas can trigger a 400 — shorten property names, reduce enum size, or flatten nested arrays.

Self-hosted / open models

Pass the schema at the serving layer, not in the prompt. vLLM and SGLang accept guided_json (JSON Schema), guided_grammar (EBNF/GBNF), or guided_regex. XGrammar (the default backend for vLLM/SGLang) compiles JSON Schema to a pushdown automaton with low per-token overhead; Outlines (dottxt-ai) uses FSM-based masking but does not support recursive schemas — use XGrammar or llguidance for those. llama.cpp uses GBNF grammars.

Provider comparison

Common failure modes

• API rejected (400) — schema uses unsupported features (default, numeric ranges, recursive $ref). Strip unsupported keywords (most SDKs do this automatically).
• Output truncated mid-object — stop_reason: "max_tokens"; the schema cannot prevent an early stop. Always check the stop reason; raise max_tokens or simplify the schema.
• Schema ignored despite no error — you used json_object instead of json_schema strict.
• Model refuses — a safety refusal overrides the schema (stop_reason: "refusal"); treat it as a first-class error branch.
• First request slow — grammar compilation; warm up with a test call and avoid schema churn.

SDK helpers

OpenAI client.beta.chat.completions.parse(response_format=PydanticModel) (Python) / zodResponseFormat(schema, name) (TS); Anthropic messages.parse(output_format=PydanticModel) (on messages.create itself the canonical parameter is output_config.format — the top-level output_format create-param is deprecated); Gemini (google-genai) accepts a Pydantic class directly as response_schema. These auto-derive the schema and parse the result.

Verified sources

• Anthropic structured outputs docs: https://platform.claude.com/docs/en/build-with-claude/structured-outputs
• Google Gemini structured output docs: https://ai.google.dev/gemini-api/docs/structured-output
• OpenAI Structured Outputs guide: https://developers.openai.com/api/docs/guides/structured-outputs
• XGrammar paper (constrained generation, arXiv:2411.15100): https://arxiv.org/abs/2411.15100
• Outlines (dottxt-ai): https://github.com/dottxt-ai/outlines

#structured-outputs #json-mode #response-format #json-schema #constrained-decoding #agents

Category: Guide