Skip to content
Multigrid

Structured output & function calling

How to make a next-token predictor emit JSON your program can rely on — what each enforcement mechanism actually guarantees, and what you still have to check yourself.

A language model does not produce data structures. It produces tokens, and JSON is a shape those tokens sometimes take. Everything in this cluster is about narrowing the gap between those two facts: token masking at the sampler, schemas the model can fill without guessing, validators that catch what the sampler cannot, and tests that hold for a function which is allowed to answer differently twice.

The single most useful idea here is also the least intuitive one. Strong enforcement does not remove errors; it relocates them. Once a grammar makes malformed JSON unreachable, the mistakes that used to arrive as a parse exception arrive instead as a well-formed object with the wrong value in it — quieter, later, and further from the code that caused them. Several pages here are about finding those.

Getting Reliable JSON Out of an LLM

The four ways to get JSON from a model, what each one actually guarantees, and the failure modes that survive all of them.

4 min read

JSON Mode vs Structured Outputs vs Grammar Constraints

Three mechanisms with three different promises, routinely discussed as if they were one feature with three names.

4 min read

Constrained Decoding: How Schema Enforcement Actually Works

The mask applied to the logits before sampling, why it is nearly free at run time, and the two costs it does impose.

4 min read

Designing a JSON Schema an LLM Can Fill Correctly

A schema is a prompt with a type system attached. Field names, field order and descriptions do more work than the types do.

5 min read

Optional Fields, Nulls and Unions: Where Schemas Break

Four JSON Schema constructs that either get rejected by strict modes or survive into generation with unclear semantics, and what to write instead.

4 min read

Enums vs Free Text: Constraining the Answer Space

What closing the output set removes by construction, and the six label-design mistakes that put the errors straight back.

4 min read

Extracting Structured Data From Messy Documents

A complete extraction pipeline — chunking, schema-constrained calls, span grounding, validation and one bounded repair.

5 min read

Validation and Repair: What to Do With Malformed Output

A cost model for choosing between local repair, a repair call and a full retry, plus the rule that keeps repair from becoming a loop.

5 min read

Streaming Structured Output: Parsing Incomplete JSON

A partial-JSON parser that turns any prefix of a stream into the largest valid object it can, so a UI can render before the response ends.

6 min read

Pydantic, Zod and Typed LLM Outputs

One type definition that produces the request schema, the runtime validation and the static type, and the three places that chain leaks.

5 min read

Function Calling vs Structured Output: Pick the Right One

The same constrained-decoding machinery under two APIs that differ in who decides whether the model answers at all.

4 min read

Nested and Recursive Schemas: The Depth Limit Nobody Documents

Where hard schema limits actually bite, a probe that finds yours, and why deep nesting fails as wrong data rather than as an error.

5 min read

Arrays and Counts: Why Models Return Seven of Ten Items

The most common extraction bug, its four distinct causes, and the fix for each — starting with the two that are not the model's fault.

5 min read

Confidence Scores in Structured Extraction

Why a confidence field the model writes is not a probability, three signals that are grounded in something, and how to check calibration yourself.

5 min read

Classification as Structured Output

One output token, a logprob read-off and a softmax over your label set — a classifier that returns a distribution instead of a word.

5 min read

Schema Versioning When Your Extraction Changes

Extracted records are derived data with a producer, and the producer changes. A migration chain, and the one field that decides whether re-extraction is possible at all.

5 min read

Structured Output Support: Test It Yourself

A script that reports what your endpoint and model actually support today, because every published capability table is out of date.

6 min read

Tool Schemas That Cost You Tokens

Tool definitions are prompt tokens on every turn. How to measure yours exactly, and the trims that matter in the order they matter.

5 min read

The Silent Drop: Parameters APIs Ignore Instead of Rejecting

A probe that tells you which request parameters your endpoint honours, which it rejects, and which it accepts and throws away.

6 min read

Testing Structured Output: A Test Suite Design

Assertions that hold for a function allowed to answer differently twice: invariants, metamorphic relations, synthesised ground truth and a flakiness budget.

5 min read

Other topics

Structured output & function calling · Multigrid