Migrating Response Format Enforcement ("json_object") Between Providers

You changed the base URL, the request still carries "response_format": {"type": "json_object"}, and the API rejected it. The fix is not one workaround, because “JSON mode” names three different guarantees and you need to know which one your parser was relying on.

The error

The rejection arrives as a 400 with an error envelope naming the offending parameter. In the OpenAI-compatible shape that is:

{
  "error": {
    "message": "Unrecognized request argument supplied: response_format",
    "type": "invalid_request_error",
    "param": "response_format",
    "code": null
  }
}

The exact message text varies by provider and by gateway — some say the argument is unrecognised, some say the value is unsupported for this model, some name the nested type rather than the parameter. The field to read is param, not message. Two nastier variants exist and are worth checking for explicitly:

• Accepted and ignored. A tolerant server that drops unknown fields returns 200 and prose. Nothing errors; your JSON parser throws downstream on a response that begins “Here is the JSON you asked for:”. If you did not get an error but did get a parse failure, this is why.
• Model-specific support. On some providers the parameter is valid but only for certain models, so the same request succeeds and fails depending on the model string — which makes it look intermittent when it is deterministic per route.

The other thing to know before you write a workaround: on the API where json_object originated, it also carries a requirement that the word JSON appear somewhere in your messages, and requests without it are rejected. If your prompt satisfied that requirement with a single throwaway sentence, that sentence is now doing all the work on the new provider and it is not enough.

Three different guarantees

These get conflated constantly, and the migration is much easier once they are separated. The general treatment is in JSON mode versus structured outputs; what matters here is which one your code assumed.

• Syntactic validity only. The output parses as JSON. Nothing about which keys it has. This is what json_object gives you, and it is a weaker promise than most people think — a valid empty object satisfies it.
• Schema conformance. The output validates against a schema you supplied, usually via a JSON Schema response format with a strict flag. Different parameter, different guarantee, and covered in the JSON schema response format.
• Semantic correctness. The values are right. No API parameter provides this and none claims to. If your migration “broke JSON mode” and what actually broke was field values, no response format setting was ever protecting you.

Fallbacks that reproduce each

Forced tool call — the closest equivalent

On a provider with tool calling but no response-format parameter, the standard substitute is to declare one tool whose input schema is your desired output shape and force the model to call it. The arguments come back as a structured object, and on APIs where a tool call’s input is already parsed you skip the JSON decode entirely.

tools = [{
  "name": "emit_result",
  "description": "Return the extraction result. Always call this tool.",
  "input_schema": {
    "type": "object",
    "properties": {
      "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
      "topics": {"type": "array", "items": {"type": "string"}}
    },
    "required": ["sentiment", "topics"]
  }
}]
tool_choice = {"type": "tool", "name": "emit_result"}

This gets you closer to schema conformance than json_object ever did, which is worth noticing: the migration can end up with a stronger guarantee than the thing it replaced. The forcing value differs by API — see mapping the tool_choice parameter. Two costs: the tool schema occupies context on every request, and some models narrate before calling, so your parser must read the call rather than the text.

Assistant prefill

Where the API lets you end the request with a partial assistant turn, seeding it with an opening brace removes the preamble problem structurally — the model has already “started” the JSON and cannot say “Sure, here you go” first. Remember to prepend the seeded character to the response before parsing, since it is not echoed back. Pair it with a stop sequence on the closing delimiter if you want to bound trailing chatter.

Prompt-only, with the constraints spelled out

Where neither is available, the prompt does the work, and it works better with four things stated explicitly rather than one: the exact schema as a literal example rather than a description; an instruction that the entire response is the JSON document; a prohibition on code fences; and one worked example of input to output. A single “respond in JSON” is the version that fails.

Set a stop sequence where you can, and set max_tokens generously — truncated JSON is the failure that looks like a model problem and is a budget problem. Its symptoms are covered in testing max_tokens truncation of JSON.

The validate-and-repair loop

Whichever fallback you use, the migration’s real deliverable is that your client stops assuming validity. Build this once and the question of which provider supports which parameter becomes an optimisation rather than a blocker.

1. Extract before parsing. Strip code fences, take the substring from the first opening brace to the last closing brace, and trim. This alone recovers most failures on prompt-only fallbacks.
2. Parse, and on failure record why. Truncation, trailing prose, a fence, a smart quote and an unescaped newline are different bugs with different fixes, and an aggregate failure count tells you none of them.
3. Validate against the schema, always. Even where the provider claims conformance. This is what makes the two providers comparable and what turns a capability difference into a number.
4. Repair with one bounded retry that includes the invalid output and the validator’s error message, and asks only for the corrected document. One retry, not a loop — a model that failed twice on the same input will usually fail a third time and you are paying for it.
5. Fail explicitly after the retry. A null result your pipeline handles beats a partially-parsed object it does not.

Cutover checklist

Before you route production traffic, confirm four things. That the parse-failure rate on a replay of recorded inputs is at or below what the old provider produced — measured on your own traffic, not assumed. That the schema-validation failure rate is separately tracked, since a valid-JSON-wrong-shape response is invisible in a parse rate. That truncation is instrumented via the stop reason rather than inferred from a parse error. And that the repair retry is capped and its cost is visible, because an unbounded repair loop on a bad prompt is the most expensive failure mode in this whole area.

Which providers offer which of the three guarantees changes with almost every model release, so treat any list you write down as dated — structured output support is the page to check rather than a constant in your code.