Parsing Model Output Safely

You asked for JSON and got JSON wrapped in a code fence, preceded by “Sure! Here is the JSON:”, with a trailing comma. A parser for model output has to survive all of that — and, more importantly, has to be unable to turn a failure into an empty result that the rest of your program treats as an answer.

The six shapes model output arrives in

Even with a system prompt that says “respond with JSON only”, these are what a parser meets in practice. The first two are common enough that any parser without a case for them is broken.

1. bare object          {"label": "billing", "confidence": 0.9}

2. fenced               ```json
                        {"label": "billing"}
                        ```

3. prose then object     Sure! Here is the result:
                         {"label": "billing"}

4. truncated             {"label": "billing", "confid
                         (finish_reason == "length")

5. two objects           {"label": "billing"}
                         {"label": "refunds"}

6. correct JSON,         {"label": "Billing", "confidence": "high"}
   wrong types           (a string where the schema said a number)

Shapes 1 to 3 are formatting and can be repaired mechanically. Shape 4 is not a parse problem at all — it is max_tokens, and the fix is upstream. Shape 5 means your prompt asked for one thing and the model gave a list. Shape 6 is a validation problem and belongs to Pydantic, as your output contract, not to the text parser.

Stripping the fence

Handle the fence first, because it is the most frequent and the easiest. The rules that matter: the opening fence may carry a language tag (```json, ```JSON, or nothing), the closing fence may be missing when the answer was truncated, and there may be text on either side.

# parsing.py
import re

FENCE_RE = re.compile(
    r"```[ \t]*([A-Za-z0-9_+-]*)[ \t]*\r?\n(.*?)(?:\r?\n)?```",
    re.DOTALL,
)


def strip_fence(text: str) -> str:
    """Return the contents of the first fenced block, or the text unchanged."""
    match = FENCE_RE.search(text)
    if match:
        return match.group(2).strip()
    # unterminated fence: the model was cut off mid-block
    if text.lstrip().startswith("```"):
        body = text.lstrip()[3:]
        body = body.split("\n", 1)[1] if "\n" in body else ""
        return body.strip()
    return text.strip()

re.DOTALL is what makes . match newlines, and the non-greedy (.*?) is what stops the pattern swallowing two blocks into one when the model emits a second example. Both are easy to leave out and both fail only on longer outputs, which is the worst way for a bug to be distributed.

Finding the object when there is prose around it

For shape 3, the naive approach — take everything from the first { to the last } — works surprisingly often and fails badly when the prose itself contains a brace. A short bracket scanner that respects string literals is barely longer and does not have that failure:

def find_json_span(text: str) -> str | None:
    """Return the first balanced {...} or [...] span, ignoring braces in strings."""
    starts = {"{": "}", "[": "]"}
    for i, char in enumerate(text):
        if char not in starts:
            continue
        closing = starts[char]
        depth = 0
        in_string = False
        escaped = False
        for j in range(i, len(text)):
            c = text[j]
            if in_string:
                if escaped:
                    escaped = False
                elif c == "\\":
                    escaped = True
                elif c == '"':
                    in_string = False
                continue
            if c == '"':
                in_string = True
            elif c == char:
                depth += 1
            elif c == closing:
                depth -= 1
                if depth == 0:
                    return text[i:j + 1]
        return None      # opened but never closed: truncated output
    return None

It returns None rather than a best guess, which is the point. A truncated object is not repairable — you cannot know whether "confid was going to be confidence or something else — and any code that pretends otherwise is inventing data.

Failing loudly

Here is the whole parser. Note that it has exactly two outcomes: a value, or an exception carrying the raw text. There is no third path that returns {}.

import json
from typing import Any


class ParseFailure(Exception):
    def __init__(self, reason: str, raw: str):
        super().__init__(f"{reason} (raw: {raw[:200]!r})")
        self.reason = reason
        self.raw = raw


def parse_json_output(text: str) -> Any:
    if not text or not text.strip():
        raise ParseFailure("empty response", text or "")

    candidate = strip_fence(text)
    try:
        return json.loads(candidate)
    except json.JSONDecodeError:
        pass

    span = find_json_span(candidate)
    if span is None:
        raise ParseFailure("no balanced JSON value found", text)
    try:
        return json.loads(span)
    except json.JSONDecodeError as exc:
        raise ParseFailure(f"invalid JSON: {exc.msg} at char {exc.pos}", text) from exc

Called with the whole model response, plus a check on finish_reason so that a truncation is reported as a truncation rather than as bad JSON:

choice = body["choices"][0]
if choice.get("finish_reason") == "length":
    raise ParseFailure("output truncated at max_tokens", choice["message"]["content"])
value = parse_json_output(choice["message"]["content"])

The reason this matters more than it looks: a parser that returns {} or None on failure moves the error to whichever part of your program first does something odd with an empty dict, often hours later and in a different service. A parser that raises tells you the model returned something unusable, on the line where the model returned it, with the text attached. Every downstream debugging session you avoid is paid for here.

Re-asking with the error attached

One repair attempt, with the parser’s complaint pasted back to the model, fixes most formatting failures. One, not a loop — if the second attempt also fails, the prompt or the schema is wrong and more attempts only spend money.

REPAIR_TEMPLATE = (
    "Your previous reply could not be parsed as JSON.\n"
    "Error: {error}\n"
    "Previous reply:\n{previous}\n\n"
    "Reply again with the JSON only. No prose, no code fence."
)


def parse_with_one_repair(call, messages: list[dict]) -> Any:
    body = call(messages)
    text = body["choices"][0]["message"]["content"]
    try:
        return parse_json_output(text)
    except ParseFailure as first:
        repair = messages + [
            {"role": "assistant", "content": text},
            {"role": "user", "content": REPAIR_TEMPLATE.format(
                error=first.reason, previous=text[:2000])},
        ]
        body = call(repair)
        return parse_json_output(body["choices"][0]["message"]["content"])

Count the repairs. A repair rate that is climbing is a signal about the prompt or a model that changed under you, and it is the sort of thing that stays invisible until somebody looks at the bill.

The outputs that are not JSON

JSON is the default answer and often the wrong one. For a single label, a yes or no, or a number, asking for JSON adds a wrapper the model can get wrong and buys nothing. Parse the smaller thing properly instead.

# parse_simple.py
import re
from typing import Sequence

LABELS = ("billing", "technical", "account", "other")


def parse_label(text: str, allowed: Sequence[str] = LABELS) -> str:
    """Accept the label despite case, punctuation and a short preamble."""
    cleaned = text.strip().strip(".\"'").lower()
    if cleaned in allowed:
        return cleaned
    # a model that said "Category: billing" or "The answer is billing."
    found = [label for label in allowed
             if re.search(rf"\b{re.escape(label)}\b", cleaned)]
    if len(found) == 1:
        return found[0]
    raise ParseFailure(
        f"no label from {allowed} in reply" if not found
        else f"ambiguous: matched {found}", text)


NUMBER_RE = re.compile(r"-?\d+(?:\.\d+)?")


def parse_number(text: str) -> float:
    matches = NUMBER_RE.findall(text.replace(",", ""))
    if len(matches) != 1:
        raise ParseFailure(f"expected one number, found {len(matches)}", text)
    return float(matches[0])

The rule both functions share: ambiguity is a failure, not a tie-break. Two labels in the reply means the model did not decide, and taking the first one invents a decision it did not make. Two numbers usually means it showed its working, which is a different answer from the one you asked for. Raising on both is what keeps the real error rate visible instead of burying it in your data.

The single-label form is also cheaper in a way worth knowing. With max_tokens set to about five, a classifier cannot produce prose even if it wants to, and the reply costs a handful of output tokens rather than forty. Combined with a closed label set in the prompt, that is usually a better design than a JSON object with one key in it.

What to do with a failure you cannot repair

The repair attempt failed. The parser raised. Something now has to decide, and the decision belongs to the caller rather than to the parser — which is the reason parse_json_output raises instead of choosing for you.

Whichever applies, the failure has to be counted. A parse failure rate is one of the most useful single numbers in an LLM system: it moves when the prompt changes, when the model is updated underneath you, and when the input distribution shifts — and it is much cheaper to watch than a quality evaluation. Emit it from the exception handler with the caller name attached, and it becomes a graph rather than an anecdote.

What not to do

• Do not repair with text.replace("```json", ""). It leaves the closing fence, breaks on ```JSON, and quietly corrupts any output that legitimately contains the string. The regex above is not much longer and is correct.
• Do not strip trailing commas with a regex. ,\s*[}\]] also matches inside string values, so a comma in the model’s prose gets deleted from your data. If trailing commas are frequent, use a tolerant parser such as json5 deliberately, rather than mangling text.
• Do not catch the exception and continue. except Exception: return {} around a parse is the line that turns a visible outage into a slow data-quality problem nobody traces back.
• Do not build this when the endpoint supports strict structured output. Where a provider offers a JSON-schema-constrained mode, the model cannot emit a fence or prose in the first place, and this whole page becomes a fallback path. JSON mode versus structured outputs explains the difference between the two things providers call by similar names.