Output Formatting: Consistent Shapes Without JSON Mode

Where a model supports a response schema, use it and stop reading. This page is for everything else: older models, open weights behind a minimal server, and providers whose structured-output flag is accepted and quietly ignored.

Choose the format for the payload

JSON is the default and it is the worst choice for exactly one common case, which happens to be the case most people have: long free text inside a field. Every quote and newline in that text must be escaped, the model has to maintain the escaping across hundreds of tokens, and one unescaped quote invalidates the entire document — including the fields that were fine.

It is worth seeing that failure concretely. A model writing a four-hundred-word summary inside a JSON string must emit \n rather than a newline and \" rather than a quote, for hundreds of consecutive tokens, while also composing the content. When it slips, the parser reports something like Invalid control character at: line 1 column 214 — a real newline inside a string — and every other field in the object is lost with it. Tagged sections carry no escaping requirement at all, which is why they degrade so much more gracefully.

The token argument is worth making concrete: fifty records with five fields in JSON pays the key names fifty times. In TSV the header is paid once. On long lists that is a real fraction of the output bill, and output tokens are the expensive side.

Prompt-side tactics that hold

• One exemplar of the exact bytes. Not a description of the schema — the literal output, including whether there is a newline at the end. This is the single highest-value line in the prompt.
• Prefill the assistant turn where the API allows it. Starting the model’s own message with { removes the preamble problem mechanically instead of asking for it to be skipped. Remember to prepend the prefilled characters back onto the response before parsing.
• Stop sequences for the epilogue. If the model reliably follows the object with “Let me know if”, stop on a string that catches it rather than adding another prohibition.
• State the empty and unknown cases explicitly. “If a field is unknown use null” prevents the invented placeholder, which is the failure that gets past validation and into the database.
• Put the contract last. Immediately before generation begins, where it is the most recent thing in context.

Field order inside the exemplar is a design decision rather than a formatting one. Generation runs left to right, so a field written early cannot be informed by one written later. Put the evidence, the extracted spans and any reasoning before the verdict and the verdict is conditioned on them; put the verdict first and everything after it is commentary on a decision that was already made.

Parse tolerantly

Assume the response is not clean and recover deterministically before giving up. Roughly this order:

def extract_json(text):
    s = text.strip()

    # 1. markdown fences — by far the most common contamination
    if s.startswith("```"):
        s = s.split("\n", 1)[1].rsplit("```", 1)[0].strip()

    # 2. prose before or after: take the first balanced {...}
    start = s.find("{")
    if start == -1:
        raise NoObjectFound(text)
    depth, in_str, esc = 0, False, False
    for i, ch in enumerate(s[start:], start):
        if in_str:
            if esc:        esc = False
            elif ch == "\\": esc = True
            elif ch == '"': in_str = False
        elif ch == '"':    in_str = True
        elif ch == "{":    depth += 1
        elif ch == "}":
            depth -= 1
            if depth == 0:
                return json.loads(s[start:i + 1])
    raise UnbalancedObject(text)      # usually a max_tokens truncation

The brace counter is what separates this from a regex: it is string-aware, so a } inside a prose field does not end the object early. And the unbalanced case is diagnostic — an object that never closes almost always means the response was truncated, so check finish_reason before you blame the prompt.

If the output is a list, prefer one object per line and parse line by line. You get partial results from a truncated response, you can start processing before generation finishes, and a single malformed line is skippable — three properties a single large array does not have, and all three matter more as the list gets longer.

The retry that actually works

Validate the parsed object against a schema — Pydantic, zod, whatever your language offers — and when it fails, feed the validation error back. The error text is unusually good prompt material because it is specific, mechanical and names the field:

ValidationError: 1 validation error for Triage
category
  Input should be 'refund', 'technical', 'billing' or 'other'
  [type=enum, input_value='refund_request', input_type=str]

-> append as a user turn:
   "Your last reply failed validation:
    <error>...the text above...</error>
    Return the corrected JSON object only."

• Keep the failed output in the conversation. The model needs to see what it produced to correct it.
• Cap retries at two, then fall back to a safe default and record the case. Unbounded repair loops turn a formatting bug into an outage with a bill.
• Do not raise the temperature to “shake it loose”. If the first attempt was malformed, more entropy makes the second worse.
• Log every repair. A rising repair rate is the earliest signal that a provider silently changed something underneath you.

Parse rate is a production metric

Track first-attempt parse rate, post-repair parse rate and repair cost per thousand requests, and alert on the first. It moves before anything user-visible does, and it is the metric that catches a model version change, a provider swap or a prompt edit that shipped without an eval — all of which look identical from the user’s side until they do not.

Alert on a rolling first-attempt rate rather than a daily average, and store the raw response text for every failure. Those failures are cheap to keep, they are the only material from which a format change can be diagnosed, and by the time somebody asks what happened, the request that started it will be several days old.