Shimming Structured Output When a Provider Has No Native Support

You have a JSON Schema, code downstream that assumes it, and a provider that will not enforce it. The shim is a loop: instruct, extract, validate, repair, bound. Every one of those five words hides a failure mode, and the loop is only useful if you handle them.

Three tiers of guarantee

Before writing anything, be honest about which tier you are moving between, because it decides what your downstream code is allowed to assume.

• Constrained decoding. The sampler is restricted at each step to tokens consistent with the schema. Invalid output is not improbable, it is unreachable. OpenAI documents this under response_format with type json_schema and a strict flag, along with the schema subset it supports (OpenAI, Structured Outputs).
• JSON mode. The output is guaranteed to parse as JSON and nothing more. Field names, types, required-ness and enum values are all still on you. This library’s JSON mode versus structured outputs page is the treatment of that difference.
• Nothing. The provider returns text. Everything is on you, and this page is about that case.

There is a fourth route worth checking before you build anything: tool-use forcing. If the provider supports tools but not response_format, declare exactly one tool whose input schema is your output schema and force it. Anthropic documents tool_choice with a type of tool and a tool name for this (Anthropic, tool use). The arguments come back as a structure rather than as prose, which removes the extraction problem entirely and is a large improvement. It does not remove validation: the arguments are the model’s idea of your schema unless the provider also constrains decoding over it. Try this first; fall back to the text shim only where tools are absent too.

Turning the schema into prompt

The instinct is to paste the JSON Schema into the system prompt. That works but is wasteful and vague, because the schema’s vocabulary is not the model’s. Two things carry far more weight than the schema text: one complete example instance, and an explicit statement of the output envelope.

function schemaPrompt(schema, example) {
  return [
    "Reply with a single JSON object and nothing else.",
    "Do not wrap it in a code fence. Do not write a sentence before or after it.",
    "",
    "The object must match this JSON Schema:",
    JSON.stringify(schema),
    "",
    "A valid example of the shape (values are illustrative, not answers):",
    JSON.stringify(example),
    "",
    "If a value is unknown, use null. Never invent a value to fill a field.",
    "Never add fields that are not in the schema.",
  ].join("\n");
}

Three of those lines exist because of specific behaviours. “Do not wrap it in a code fence” reduces but does not eliminate fenced output, which is why the extractor below still strips fences. The null instruction matters because the alternative to a null is not an omission — it is a plausible fabrication, and a required field is a standing invitation to invent one. The line about extra fields matters because a model that adds a helpful notes key will fail a validator configured with additionalProperties: false, and you would rather it did not add the key than relax the validator.

Keep the schema small. Deeply nested objects, long enums and unions are where the compliance rate falls, and on a provider with no constraint you are paying that rate directly. Two calls each returning a flat object beat one call returning a tree, and they are individually repairable.

Getting JSON out of the reply

The single most common bug in a hand-rolled shim is calling JSON.parse on the raw content and treating the throw as a model failure. Most of the time it is a wrapping problem, and it is recoverable without another call. Order matters here: check for truncation before attempting to parse, because truncated JSON and refused JSON need opposite responses.

// 1. Truncation first. If generation stopped at the token limit the JSON is
//    cut mid-structure; parsing it is pointless and repairing it is wrong —
//    you must re-ask with a larger limit or a smaller schema.
//    OpenAI-shaped responses: choices[0].finish_reason === "length".
//    Anthropic-shaped responses: stop_reason === "max_tokens".
if (truncated(response)) throw new OutputTruncated();

// 2. Strip a code fence if there is one.
function unfence(text) {
  const fence = text.match(/```(?:json)?\s*([\s\S]*?)\s*```/i);
  return fence ? fence[1] : text;
}

// 3. Take the outermost balanced object, ignoring braces inside strings.
function firstJsonObject(text) {
  const start = text.indexOf("{");
  if (start === -1) return null;
  let depth = 0, inStr = false, esc = false;
  for (let i = start; i < text.length; i++) {
    const ch = text[i];
    if (esc) { esc = false; continue; }
    if (ch === "\\" && inStr) { esc = true; continue; }
    if (ch === '"') { inStr = !inStr; continue; }
    if (inStr) continue;
    if (ch === "{") depth++;
    else if (ch === "}" && --depth === 0) return text.slice(start, i + 1);
  }
  return null; // opened and never closed: treat as truncation
}

The string-aware scan is not fastidiousness. A naive search for the last closing brace breaks the moment a field value legitimately contains one, which happens constantly on any task that extracts from source code, templates or log lines. Getting this wrong produces a shim that works for months and then fails only on the inputs that mention JSON.

Once you have a candidate string, parse it and validate against the schema with a real validator — Ajv in TypeScript, Pydantic or jsonschema in Python. Do not hand-write field checks. The reason is the next section: you are going to feed the validator’s error messages back to the model, and a real validator produces messages with a path and a reason, which are exactly what a repair turn needs.

The repair turn

When validation fails, the cheap and effective move is to continue the conversation rather than start it again. Append the model’s invalid reply as an assistant turn, then a user turn containing the validator’s errors verbatim.

const errors = validate.errors
  .map(e => `${e.instancePath || "/"}: ${e.message}`)
  .join("\n");

messages.push({ role: "assistant", content: rawReply });
messages.push({
  role: "user",
  content:
    "That reply did not validate against the schema. Errors:\n" + errors +
    "\n\nReply again with the corrected JSON object only.",
});

This works better than re-asking from scratch because the failure is usually local — one enum value, one field typed as a string that should be a number — and the model can see both its own output and the specific complaint. It costs a full prefill of the growing conversation, so bound it hard: two repair attempts, then give up and surface the failure to the caller with the last raw reply attached. A loop that keeps trying is how a structured-output shim turns a bad minute into a large bill.

One thing not to do: raise the temperature on retry. The reasoning sounds plausible — at temperature 0 the same input yields the same output, so vary it to escape. But the repair turn has already changed the input; the conversation now contains the failed reply and the error list, so the distribution is different regardless. Raising temperature only makes the retry less likely to follow the schema, which is the one thing you are asking it to do. Keep the sampling settings identical and let the added context do the work.

Failure modes you will see

• Fenced output. Handled by the extractor above. Very common, entirely benign, and not worth a retry.
• Commentary around the object. “Here is the JSON you asked for:” before, or a summary after. Also handled by extraction.
• Truncation at the token limit. The reply is valid up to the point it stops. Detect it from the finish reason before parsing, and respond by raising the output limit or splitting the schema — never by attempting to close the braces yourself, which produces a document that parses and is missing data nobody notices is gone. This library’s test for max-token truncation of JSON is the regression that keeps it caught.
• Invented enum members. A schema with enum: ["low", "medium", "high"] gets "moderate". This is the failure most improved by listing the permitted values in the prompt in addition to the schema, and the one a repair turn fixes most reliably.
• Numbers as strings. "42" where the schema says integer. Tempting to coerce silently. Do not: coercion hides the fact that the model is not following the schema, and the same latitude will produce a string where you needed a boolean and"false" is truthy.
• Refusal instead of JSON. On a safety-triggering input you get a sentence, not a document. A refusal is not a schema failure and repairing it wastes two more calls. Detect the case where no object was found at all and route it separately.
• Valid JSON, wrong content. The schema says every field is present and typed; nothing says the values are right. A shim restores structure, never accuracy.

Build it

1. Write the JSON Schema you actually need, then delete from it every field the downstream code does not read. Compliance falls with schema size and you are paying that rate yourself.
2. Probe the route. Send one request with the provider’s native structured-output parameter and a schema requiring a single integer field, with a prompt that invites a sentence. If you get the integer, stop — you do not need a shim.
3. If tools are supported, declare one tool whose input schema is your schema and force it with the provider’s equivalent of a named tool_choice. Read the arguments off the tool call. Skip to step 6.
4. Otherwise build the prompt with schemaPrompt above: envelope instruction, schema, one example instance, the null rule, the no-extra-fields rule.
5. Set the output token limit generously — at least twice your expected document — because truncation is the failure that costs a whole extra call to discover.
6. Check the finish reason first, then unfence, then take the outermost balanced object, then parse, then validate with Ajv or Pydantic.
7. On a validation failure, append the raw reply and the validator errors and re-ask. Maximum two repairs, same sampling parameters, and count the repairs in your metrics — a rising repair rate is the first visible sign of a silent model update.
8. On final failure, throw with the last raw reply attached, and make sure the caller has somewhere to put a request that did not produce a usable object. A shim without a defined giving-up behaviour is a shim that retries forever.