Constrained Generation With llama.cpp Grammars (GBNF)

A grammar does not ask the model for JSON. It removes every token that would break JSON from consideration before the sampler picks, which is why it works on a 1B model that has never been fine-tuned for structured output.

Constraint happens at the sampler, not the model

At each step the model produces a logit for every token in the vocabulary. llama.cpp’s grammar sampler holds a parser state, works out which tokens could legally continue the string given that state, and sets every other logit to negative infinity. Whatever sampling you have configured — temperature, top-p, min-p — then runs on what is left.

Three consequences follow directly, and they are the ones that decide whether a grammar is the right tool:

• Output is valid by construction, not by luck. There is no retry loop and no validation step, because an invalid continuation was never reachable.
• Validity is not correctness. Forcing a field to be a number does not make it the right number. A constrained model that does not know the answer will emit a well-formed wrong one, and it will do so confidently because you removed its ability to hedge.
• It costs time per token. The legal-token set is recomputed at every step. On a large vocabulary with a complex grammar this is measurable, and it is why a tight grammar is cheaper than a permissive one.

It is worth being clear about how this differs from the two things it is usually confused with. Prompting for JSON changes what the model wants to say and nothing else; the model can still emit a stray comma. Biasing individual tokens with logit bias makes particular tokens more or less likely without any notion of whether the string so far is well-formed. A grammar is the only one of the three that carries parser state, which is why it can guarantee a closing brace arrives and the other two cannot.

The syntax you actually need

GBNF is Backus-Naur form with regex-like extensions, documented in llama.cpp’s GBNF guide. Rules are written name ::= sequence, rule names must be lowercase and dashed, and the rule called root is the one the whole output must match.

• Literals in double quotes, character classes in brackets: [0-9], [a-zA-Z_]. Negate a class with ^, as in [^"].
• Alternatives with |, grouping with parentheses.
• Repetition with *, +, ?, and counted forms {m}, {m,} and {m,n}. The counted forms are what keep a grammar from allowing a thousand-character string where you wanted a short one.
• Comments with #. Unicode escapes as \xXX, \uXXXX and \UXXXXXXXX.
• Token matching with angle brackets — <think> matches a token whose text is exactly that, and <[1000]> matches token ID 1000. Prefix with ! to match anything except. This only parses if the string is a single token in that model’s vocabulary, so it is model-specific by nature.

A grammar that forces one JSON shape

llama.cpp ships a general grammars/json.gbnf that accepts any JSON object. That is usually not what you want: a general JSON grammar lets the model invent field names. Constrain the shape instead. Save this as ticket.gbnf — it permits exactly one object with three fields in a fixed order:

# Exactly one object: {"summary": "...", "severity": "low|medium|high", "line": N}
root ::= "{" ws
           "\"summary\":"  ws string ws "," ws
           "\"severity\":" ws severity ws "," ws
           "\"line\":"     ws integer ws
         "}"

severity ::= "\"low\"" | "\"medium\"" | "\"high\""

# A JSON string, capped so the model cannot ramble
string ::= "\"" char{1,200} "\""
char   ::= [^"\\\x7F\x00-\x1F] | "\\" (["\\bfnrt] | "u" [0-9a-fA-F]{4})

integer ::= "0" | [1-9] [0-9]{0,6}

# Optional whitespace, bounded on purpose
ws ::= | " " | "\n" [ \t]{0,20}

Three things in there are deliberate and are the difference between a grammar that works and one that hangs. The char{1,200} cap stops an unbounded string rule from letting a confused model generate forever. The bounded ws rule is copied from llama.cpp’s own JSON grammar for the same reason — an unbounded whitespace rule is a legal infinite loop and models find it. And fixing the field order shrinks the legal-token set at every step, which makes the grammar both faster and more reliable than one that permits any order.

Running it, three ways

1. From the CLI, with the file.

   llama-cli -m model.gguf \
     --grammar-file ticket.gbnf \
     -p "Summarise this stack trace as a ticket:\n<trace here>"

   --grammar takes the grammar inline as a string if you would rather not keep a file.
2. From a JSON Schema, without writing GBNF. -j/--json-schema converts a schema to a grammar for you, and -jf/--json-schema-file reads it from disk. The help text is explicit that schemas with external $refs are not handled this way and want --grammar plus examples/json_schema_to_grammar.py instead.
3. Per request, against the server. llama-server accepts a grammar field on its native completion endpoint and a json_schema in the OpenAI-shaped response_format, so different requests to one server can carry different constraints. That is normally what you want in an application, since a grammar passed at startup applies to everything.
4. Verify by asking for something the grammar forbids. A grammar that is working produces a valid object anyway; a grammar that failed to parse produces an error at startup rather than silently doing nothing, which is the good failure mode.

Where grammars stop helping

The constraint is over characters the tokenizer can produce, and that is a narrower thing than it sounds. A token that spans a boundary — one that contains both the closing quote and the following comma — is either legal in full or not available at all, so a grammar can occasionally force the model onto a less natural tokenisation of the same text. On most models this is invisible; on models with unusual vocabularies it shows up as slightly stilted string contents.

Grammars also cannot express constraints that depend on values rather than shapes. “line must be within this file’s length” is not a grammar; it is a validation step you still have to write. And a grammar that is too strict has a specific failure mode worth knowing: if the model wanted to refuse and the grammar has no path to a refusal, it will fabricate a conforming answer instead. Give the schema an explicit “unknown” branch when refusal is a legitimate outcome.

If you need the same structured output across a local model and a hosted one, note that the guarantees are different in kind: the grammar is enforced in the sampler on your machine, while hosted structured output support varies by provider and by model.