JSON Mode vs Structured Outputs vs Grammar Constraints

These three get used as synonyms in issue threads and in vendor marketing. They are not synonyms. They make three different promises, and knowing which promise you bought decides what your code still has to check.

Three guarantees, not one

All three are implemented the same way underneath — a mask applied to the logits before sampling. What differs is who writes the automaton and how expressive it is allowed to be.

JSON mode: it will parse

The constraint is the JSON grammar itself, nothing more. Your json.loads will succeed. Your record["invoice_id"] may still raise a KeyError, and frequently does when the document is unusual, because the model was free to choose the keys.

JSON mode is not obsolete. It is the right choice when you genuinely do not know the shape in advance — open-ended annotation, exploratory extraction, a “list whatever entities you find” pass whose output a human will read. It is the wrong choice the moment a downstream field access exists.

Structured outputs: it will validate

You supply a schema, the provider compiles it, and the sampler cannot leave the schema. In OpenAI’s implementation, announced 6 August 2024, the flag that makes this a guarantee rather than a hint is strict: true — without it the schema is treated as guidance and you are back to something closer to rung two.

This distinction is the single most common misconfiguration in the area: a schema is sent, the field name looks right, strict is absent or false, and the behaviour is “usually correct” rather than “always correct”. It fails a few times per thousand, which is exactly the rate at which nobody notices until a report is wrong.

There is one test that tells the two apart in a single call, and it is the only reliable one: instruct the model, in the prompt, to produce a value the schema forbids. Give it an enum of three colours and demand it set the field to a fourth. Under a real constraint the token is masked and the instruction is simply unable to take effect. Under a hint, the model obeys you and the response violates the schema you sent. Nothing in the response body distinguishes the two mechanisms on ordinary inputs, which is why this contradiction test exists and why it is the first row of the probe.

The two also differ once you stream. A hinted schema can begin coherently and go wrong at the end, so nothing rendered mid-stream can be trusted; an enforced one is a valid prefix at every point, which is what makes progressive rendering safe. That difference is invisible on a request you wait for, and it decides whether you can show the user anything before the response completes.

Grammars: it will match your language

A grammar constraint is the general case that the other two are instances of. llama.cpp reads GBNF, a BNF variant, and ships example grammars including one for JSON. vLLM offers guided decoding with a pluggable backend. The published engines in this space — Outlines and XGrammar being the two most cited — both work by compiling the grammar into a state machine and precomputing which vocabulary tokens are legal in each state.

Reach for a grammar when JSON is the wrong container: a fixed-width report, a single token from a closed set, a DSL your own parser reads, or a strict subset of SQL. Requiring the model to wrap those in JSON first is two constraint systems where one would do.

The subset problem

“Supports JSON Schema” is never true without qualification. Every hosted implementation defines a subset, and the exclusions are load-bearing. OpenAI’s strict mode documents that every property must be listed in required and every object must set additionalProperties: false; a long list of validation keywords — minLength, pattern, minItems, format and their neighbours — sits outside the supported set. Gemini’s responseSchema is defined against a subset of the OpenAPI 3.0 schema object, which is a different subset with different holes.

The behaviour when you send something outside the subset is the part that matters, and it is not consistent: some stacks reject the request with a 400 naming the offending keyword, and some accept it and drop the keyword. The second case is genuinely dangerous, because your schema says minItems: 10 and your data says otherwise and nothing anywhere reports a problem. There is a script for finding those.

Picking one

• Fixed shape, hosted model. Structured outputs with strict: true. Default choice; nothing else comes close on effort per unit of reliability.
• Fixed shape, model without schema support. Forced tool call if the API has tools; JSON mode plus a validator if it does not.
• Unknown shape. JSON mode, and treat the result as untrusted input all the way through.
• Not JSON, or self-hosted. A grammar. It is less work than it looks and removes a whole layer of encoding.

Whatever you pick, keep the validator. Schema enforcement moves the error class; it does not empty it.