Truncated Output: Diagnosing an Unfinished Answer

An answer that stops mid-sentence is not a mystery, because the API told you why. The field is in every response and almost every integration ignores it — which is how a truncation ends up being debugged as a prompt problem for two days.

Read the stop reason first

OpenAI-compatible responses carry choices[0].finish_reason; Anthropic’s carry stop_reason on the message. Same concept, different vocabulary, and both are populated on every non-streaming response. The single highest-value change most codebases can make in this area is one assertion at the boundary:

OK = {"stop", "end_turn", "tool_calls", "tool_use", "function_call", "stop_sequence"}

def unwrap(response):
    reason = (getattr(response.choices[0], "finish_reason", None)
              or getattr(response, "stop_reason", None))
    if reason is None:
        raise IncompleteResponse("no stop reason -- stream ended early")
    if reason not in OK:
        raise IncompleteResponse(f"finish_reason={reason}")
    return response.choices[0].message.content

Without it, a truncated JSON object arrives at your parser, the parser raises a syntax error, and the error you investigate is three layers away from the cause.

Put the same value in your telemetry as a dimension on every request, not only in the error path. The distribution of stop reasons over a week is one of the most informative cheap metrics available: a rising share of length means your budgets no longer fit the work, a nonzero share of content_filter means a policy surface is engaging with your traffic and nobody has looked at which prompts, and a share of missing reasons means your transport is dropping streams. All three are invisible if the field is only read when something already threw.

Every value, and what it means

The length case, in detail

Three different causes produce this one value, and they need different fixes.

• Your max_tokens is genuinely too small. The obvious case. Note that many SDKs and gateways apply a default when you omit the parameter, so “I did not set a limit” does not mean there is not one.
• Prompt plus output exceeds the context window. For most models the window is shared, so a long prompt shrinks the space available for the answer. Raising max_tokens here produces an error rather than a longer answer. The fix is on the input side.
• Reasoning tokens consumed the budget. The one that surprises people. On reasoning models, internal thinking tokens are billed as output and count against the same limit, so a model can spend the entire budget reasoning and return an empty or near-empty visible answer with a length stop reason. If you see empty content plus a nonzero output-token count, this is what happened. Raise the budget substantially, or lower the reasoning effort where the API exposes that control.

And one non-cause that is worth ruling out: a repetition loop that ran until the limit. The stop reason says length, the actual bug is on the repetition loops page, and raising the budget makes the bill larger without making the answer better.

The silent one: streams that just stop

The failure with no error attached. In server-sent-events streaming the response is a sequence of chunks and completion is signalled by a final chunk carrying a finish reason (and, in the OpenAI dialect, a [DONE] sentinel). If the connection drops, an intermediate proxy times out, a load balancer closes an idle connection, or the client library exits its loop on an exception it swallowed, your stream simply ends. The text you accumulated looks like a short answer. Nothing raises.

Guard it explicitly: track whether a terminal event was observed and treat its absence as a failure. Common causes are worth knowing — proxy and load-balancer idle timeouts shorter than your longest generation, buffering layers that hold chunks until a timeout, and serverless platforms with a hard response-duration cap. If truncation correlates with long answers and disappears when you call the provider directly, the problem is in your own infrastructure and no model parameter will fix it.

The reasoning-model variant of this deserves its own note, because it looks like a hang rather than a truncation. A model that thinks for thirty seconds before emitting a visible token sends nothing during that window, and an idle-connection timeout tuned for a chat stream will close the connection before the first content chunk arrives. From the client’s side the request simply ends with nothing in it. Set the timeout against the time to first visible token, not against a typical streaming interval, and prefer a provider stream that emits keepalive events over one that does not.

One more asymmetry to design around: with streaming, a partial answer has already reached the user by the time you detect the problem. You cannot silently retry and replace it. Either buffer complete responses before rendering — losing the latency benefit that streaming exists for — or render progressively and have a defined way to append a correction or mark the answer as incomplete. Deciding this after shipping usually means the user sees half an answer and no indication that it is half.

Handling truncation without making it worse

• Never repair truncated JSON with string surgery. Appending closing braces produces a parseable object with missing data, which is far worse than a parse error because it fails silently downstream. Retry, or fail.
• Continue rather than regenerate, where the task allows it. Send the partial output back with an instruction to continue from exactly where it stopped. Cheaper than regenerating, though it risks a seam; not viable for structured output, where you should retry with a larger budget.
• Ask for less, not for more budget. Chronic truncation usually means the task is too big for one call. Splitting it produces better answers as well as complete ones — a long generation degrades in quality towards the end for the same self-conditioning reasons as everything else in this cluster.
• Set the budget from a measured distribution. Log output token counts, take the 99th percentile, add headroom. A default copied from an example in a tutorial is the most common cause of length truncations in production.
• Keep truncation out of your quality metrics. A truncated response graded as a wrong answer pollutes every number on the measurement page. Filter on the stop reason first, then grade.