Output Got Longer (or Empty) After the Model Change

“The summaries are twice as long since the switch.” That is one bug report covering two different faults, and treating it as one is why the prompt-tweaking goes round in circles.

Two symptoms that look like one

Separate them before touching the prompt. The first fault is truncation: responses are cut off, or arrive completely empty, and the terminal reason says the budget was exhausted. The second is verbosity: responses are complete, well-formed and simply longer or shorter than they used to be, with a normal terminal reason. They have nothing in common except the complaint.

The distinguishing field is the one you are probably not logging. On Chat Completions it is finish_reason — documented values stop, length, tool_calls, content_filter and the deprecated function_call. On the Messages API it is stop_reason, whose values include end_turn, max_tokens, stop_sequence, tool_use, pause_turn, refusal and model_context_window_exceeded. On the Responses API the equivalent is incomplete_details.reason, taking max_output_tokens or content_filter. Log whichever applies, per request, before you debug anything else. If the distribution of that field shifted at the cutover, you have fault one. If it did not, you have fault two.

Cause one: the budget now counts something else

The nastiest version of fault one is a response with an empty content field and a length-style terminal reason, and it is not a bug in the provider. It happens because the parameter that caps output has changed what it counts.

OpenAI’s Chat Completions API supersedes max_tokens with max_completion_tokens for reasoning-capable models, and the newer parameter budgets reasoning tokens together with the visible ones. Usage reports the split: completion_tokens_details.reasoning_tokens sits alongside completion_tokens in the response. So a call that carried over a cap of 300 — a value chosen years ago because summaries ran about 200 tokens — can now spend the entire 300 on reasoning and emit nothing. The message content is an empty string. Downstream, a validator that expects JSON reports a parse error on empty input, and the error you are handed is three layers away from the cause.

Three related budget changes cause the same family of symptom:

• The cap became mandatory. The Messages API requires max_tokens; an adapter that previously omitted the parameter has to invent a value, and whatever it invents is the new de facto length limit. If somebody picked a round number to make the request validate, that number is now your product decision.
• Stop sequences stopped firing. A prompt that relied on a sentinel string to end generation depends on the model reliably emitting that exact string. A different model emits a variant, the sequence never matches, and generation runs to the cap. The terminal reason distinguishes this cleanly: you were getting stop_sequence and are now getting the cap value.
• The cap is now the only thing limiting length. If the old model happened to finish naturally just under the cap, the cap was never really binding and nobody knew. On a more verbose model it becomes binding immediately, which is fault two wearing fault one’s clothes.

The fix for fault one is arithmetic, not prompting. Set the cap from the length you actually want plus headroom, and where reasoning tokens are counted in the same budget, add the reasoning allowance explicitly rather than hoping. Then assert on the terminal reason in your client and treat a cap-exhausted response as an error rather than as content.

Cause two: trained verbosity moved

If the terminal reason is normal and the text is simply longer, you are looking at a property of the model’s post-training. Models differ in how much they elaborate absent instruction: how readily they add a preamble restating the task, whether they close with a summarising paragraph, how often they reach for bulleted structure. None of that is documented as a number by anybody, and it should not be — it is a tendency, not a specification.

What matters is why your prompt did not constrain it. Almost always the old prompt did not actually specify a length; it specified a vibe (“concise”, “brief”) that the old model happened to interpret the way you wanted. That is not a contract, and it did not survive. A prompt that says “be concise” is portable only in the sense that it produces some length everywhere.

Recalibrating the prompt

Express the budget in a unit the model can actually count, and models cannot reliably count tokens or characters — they can count sentences, paragraphs and list items, because those are units they emit as discrete things. So:

# Weak, and the thing that broke:
Summarise the ticket concisely.

# Portable, because every constraint is countable and checkable:
Summarise the ticket in at most three sentences.
Do not restate the question. Do not add a closing summary.
Output only the summary text, with no heading and no bullet points.

Two of those lines are doing more work than the sentence limit. Explicitly forbidding the preamble and the closing summary removes the two structures that most commonly account for a doubling in length, and they are structures a model adds when it is being helpful rather than when it is being verbose. Forbidding bullets matters when the downstream consumer is a fixed-height UI element, because a bulleted answer of the same token count occupies far more vertical space.

Where the real constraint is characters rather than sentences — a database column, a UI card, an SMS — do not put the character number in the prompt and trust it. Put a countable proxy in the prompt, validate the character count in code, and retry once with a shortening instruction that includes the actual overage. That is the pattern in testing an output length character budget.

The guard that makes it not happen again

Length is the easiest model property to regression-test, and almost nobody does it, because it feels too trivial to assert. Add to the evaluation set a length distribution rather than a length assertion: run the golden inputs, record the character count of each output, and fail the suite if the median moves by more than a stated fraction or if any output exceeds the hard downstream limit. A median-based check tolerates the natural variation that a per-example assertion would flag constantly.

Two adjacent behaviours are worth adding to the same suite while you are there. Assert the distribution of the terminal-reason field, so a rise in cap-exhausted responses fails the build rather than degrading production quietly. And assert that the output contains no preamble, with a cheap check for a first line ending in a colon — the single most common shape of the extra paragraph. If your downstream consumer parses the output, an unexpected preamble is not a style issue, it is a parse failure waiting for the next deploy; see migrating a prompt chain’s output format.