Validation and Repair: What to Do With Malformed Output

“Repair or retry” is the wrong question, because there are three options and the cheapest one involves no model call at all. The decision is not about cost first — it is about what kind of thing went wrong.

Three responses, not two

Classify the failure first

Route on the failure class, not on cost. The classes and their correct response:

• Syntactic. Fence, preamble, trailing comma, unclosed brace. Local repair. Sending this to a model is paying for something str.strip does.
• Truncation — finish_reason == "length". Retry with a higher limit, or a smaller input. A repair call cannot invent the content that was never generated, and asking it to produces a plausible completion rather than the real one. This is the worst thing you can send to a repair prompt.
• Schema violation. Wrong type, missing key, value outside an enum. Repair call — the model has the content and got the container wrong, which is exactly what a short correction turn fixes. But first check whether you are using strict mode, since this class should be empty if you are.
• Semantic violation. The record validates and is wrong: a total that does not equal the line items, a date in 1970, a quote not present in the source. Retry, with the check named in the prompt. Repair here just asks the model to re-justify a conclusion it already reached, and it is very good at that.
• Refusal. Neither. Log it and route to a human. Retrying a refusal is how you end up with an automated jailbreak loop in your codebase.

The cost model

Per successful record, with your own numbers:

  P_in, P_out    price per input / output token
  I, O           input and output tokens of the original call
  p              fraction of calls needing intervention  (measure this)
  r              fraction of interventions that succeed   (measure this too)

  base    = I*P_in + O*P_out

  retry   = base + p * base                       ... second full call
  repair  = base + p * (I_r*P_in + O_r*P_out)     ... I_r = error + bad output
                                                      O_r = corrected object only

  repair is cheaper when   I_r*P_in + O_r*P_out  <  I*P_in + O*P_out

  and the interesting term is I_r: a repair call does NOT resend the
  document. It sends the schema, the bad object and the error. For any
  extraction over a long input, I_r is a small fraction of I, and that
  alone decides it.

The success rate r does not appear in the comparison above because it multiplies both branches similarly — but it does decide whether either is worth doing, and it is the number to instrument first. A repair path with r near zero is pure cost plus latency.

Worked, with hypothetical prices

These prices are invented for the arithmetic. Substitute your own; the shape of the answer is what transfers.

Suppose P_in = $1.00 and P_out = $4.00 per million tokens. A document extraction sends I = 12,000 input tokens and returns O = 600.

base   = 12000*1.00/1e6 + 600*4.00/1e6
       = 0.01200 + 0.00240            = $0.01440

full retry on failure:
         adds another $0.01440

repair call: I_r = 900 (schema + bad object + error), O_r = 600
       = 900*1.00/1e6 + 600*4.00/1e6
       = 0.00090 + 0.00240            = $0.00330

ratio  = 0.00330 / 0.01440            = 0.229  -> repair costs ~23% of a retry

at p = 3% of 200,000 calls/month = 6,000 interventions:
  all retries : 6000 * 0.01440       = $86.40
  all repairs : 6000 * 0.00330       = $19.80
  difference                          = $66.60 / month

Which is the honest conclusion: on a workload this size the cost difference is real but it is not the reason to choose. Sixty-six dollars does not justify a repair path that occasionally launders a truncated document into a plausible-looking record. Latency and blast radius decide this, and the arithmetic mostly tells you when you can stop thinking about cost — which, at these volumes, is immediately.

Rerun it with I = 100,000 (a long contract) or with an output price ten times the input price and the picture changes materially. Put the four numbers in a spreadsheet once.

The repair prompt

Short, mechanical, and explicitly forbidding the thing you fear. Do not resend the source document — that turns the repair into a second full call and reintroduces the cost you were avoiding:

system: You fix malformed JSON. Return only the corrected object.
        Change nothing except what the error requires. Do not invent
        values for fields that are absent; use null. Do not add fields.

user:   Schema:
        <schema>

        The object produced:
        <bad output verbatim>

        The validator said:
        line_items[2].unit_price: expected number, got "12,50"

        Return the corrected object.

Include the validator’s own message rather than a paraphrase. Pydantic and Zod both produce errors with a path and an expected type, and the path is the single most useful token in the repair prompt. Notice also that the example error is a European decimal comma — a repair call fixes that instance, and a line in the field description fixes every future one.

Send the repair as a fresh conversation rather than as another turn of the original one. Appending a correction to the transcript leaves the bad output in context, where it is the strongest available precedent for what the answer should look like, and models are obliging about precedent. A clean two-message request containing only the schema, the bad object and the error is both cheaper and less likely to reproduce the mistake it was sent to fix.

Rules that keep it bounded

• One repair attempt. Ever. A second attempt on the same record buys almost nothing and is how a $0.003 call becomes an unbounded loop during an incident. Fail the record and count it.
• Repair output is validated identically. No relaxed schema for the second pass. If it does not validate, it is a failure, not a partial success.
• Count by reason. A single repairs_total counter is nearly useless; the same counter labelled by failure class tells you whether to fix your schema, your chunking or your max_tokens.
• Repairs are a smell. A steady repair rate means the schema, the prompt or the enforcement mode is wrong. The goal is a repair path that exists and is almost never used.
• Never repair for a human. If the record is going to a person for review anyway, send the malformed original with the error attached. A repaired record hides the fact that something went wrong at exactly the moment someone is looking.