Getting Reliable JSON Out of an LLM

“Reliable JSON” is not one problem. It is a syntax problem, a schema problem and a truth problem, and the popular fixes solve them in that order — which is why people who solved the first one are surprised to still be losing records.

Four rungs, four guarantees

There are four ways to get JSON out of a model, and they form a ladder where each rung buys a strictly stronger guarantee at a strictly higher cost in flexibility. The mistake worth avoiding is assuming the top rung is a guarantee of correctness. It is a guarantee of shape.

Rung 1: ask, then clean up

You put “respond with JSON only” in the prompt and hope. Modern instruction-tuned models are good at this, which is exactly the trap: it works often enough that the failures arrive in production rather than in development.

The four things that break, in descending order of annoyance:

• The fence. The model wraps the object in a triple-backtick block, sometimes labelled json. Python then gives you json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0), which is the error for “the first character was not the start of a value” and reads like nonsense until you print the raw string.
• The preamble. “Sure! Here is the extracted data:” before the object, or a helpful paragraph after it.
• Trailing commas and comments. Valid JavaScript, valid JSON5, not valid JSON.
• Smart quotes. Rare, and vicious when the document being processed contains typographic quotes the model echoes.

The standard defence is a fence-stripping regex plus a substring from the first { to the last }. It is fine, it costs nothing, and you should keep it even after moving up the ladder, because it also catches the case where a provider’s constrained decoder is not actually engaged and nobody told you.

Rung 2: JSON mode

OpenAI introduced response_format: {"type": "json_object"} in November 2023, and most OpenAI-compatible endpoints copied the field. It guarantees the response parses. It says nothing at all about content — you can get {}, or your keys with the wrong types, or a completely different shape than you asked for.

It also has a documented footgun: OpenAI requires the string “json” to appear somewhere in the messages, and returns a 400 reading ‘messages’ must contain the word ‘json’ in some form, to use ‘response_format’ of type ‘json_object’ if it does not. That check exists because a model told to emit JSON and not told what JSON will happily emit an infinite run of whitespace or newlines until it hits max_tokens, and the guard rail is cheaper than the support ticket.

Rung 3: schema-constrained decoding

This is the rung most people should be on. You send a JSON Schema, the provider compiles it into a constraint on the sampler, and tokens that would break the schema are masked out before sampling — so an invalid document is not unlikely, it is unreachable. OpenAI shipped this as Structured Outputs in August 2024 (response_format with type: "json_schema" and strict: true); Google exposes a comparable thing on Gemini as responseSchema alongside responseMimeType: "application/json"; Anthropic’s long-standing route is a forced tool call, where tool_choice names a single tool and the model must fill that tool’s input schema. Check each vendor’s current docs before relying on any specific field name — this list has changed twice already.

The catch is that no provider accepts all of JSON Schema. Each supports a subset, and the parts outside it are either rejected at request time or accepted and ignored. Which constructs fall outside is worth knowing before you design the schema rather than after.

Rung 4: your own grammar

If you run the model yourself, you are not limited to JSON Schema. llama.cpp takes a GBNF grammar file; vLLM exposes guided decoding with a choice of backend; the same machinery constrains output to CSV, to a SQL dialect, to a date format, or to one of five literal strings. This is the only rung where the constraint can express something JSON Schema cannot, such as “a number between 1 and 5 followed by a newline” as a single grammar rule.

What none of them fix

Every rung above constrains the form. Four failures ignore all of them, and these are the ones that will actually cost you data:

• Truncation. A grammar cannot stop the generation hitting max_tokens mid-object. You get a valid prefix of a valid document, which is not a valid document. Always branch on finish_reason before you branch on the parse result: "length" is your bug, not the model’s.
• Refusal. Providers that implement safety refusals reserve the right to return prose instead of your schema. OpenAI surfaces this as a separate refusal field precisely so you can tell it apart from a parse failure.
• Empty-but-valid. The single most common production outcome once decoding is constrained: every required field present, every one filled with null, "" or "N/A". The schema is satisfied and the record is worthless.
• Confidently wrong values. Constrained decoding guarantees the value is a string in your enum. It does not guarantee it is the right member of the enum.

Which is the real lesson of the ladder. Climbing it converts loud failures into quiet ones. Budget the effort you save on parsing into validating the semantics, because that is where the errors went.