Testing max_tokens Truncation Doesn't Break JSON Output

SyntaxError: Unexpected end of JSON input, or in newer V8, Unterminated string in JSON at position 412. The output was valid JSON right up until the point where it stopped, and it stopped because you capped it. The fix is not a better parser.

The error you pasted into search

The traces vary by runtime and all mean the same thing. Node throws SyntaxError: Unexpected end of JSON input for an empty or abruptly ended document and Unterminated string in JSON at position N when the cut landed inside a string literal. Python raises json.decoder.JSONDecodeError: Expecting ',' delimiter or Unterminated string starting at. A schema validator downstream will report something less helpful, because it never gets a value to validate.

The tell is that it is input-dependent and looks intermittent. Short answers parse; long ones do not. If your logs show the failure correlating with the size of the requested output — more items, longer descriptions, a bigger document to summarise — you have this bug and not a model quality problem.

Why JSON mode does not save you

This is the part that surprises people. Constrained decoding guarantees that the tokens the model emits form valid JSON as it goes. It does not guarantee the model finishes. When generation stops because the token budget ran out, you get a prefix of a valid document, which is not a valid document. The constraint and the cap are enforced by different parts of the stack and neither knows about the other.

The signal that this happened is in the response and it is unambiguous. For chat completions, choices[0].finish_reason takes one of a small set of values: stop when the model ended naturally or hit a stop sequence, length when it hit the token cap, tool_calls when it stopped to call a tool, content_filter when content was filtered, and the deprecated function_call on older function-calling responses. A value of length means the document is incomplete, full stop. There is no case in which you should attempt to parse it and hope.

The guard: check finish_reason first

One function, applied everywhere you parse a model response, and the class of bug is gone.

export class TruncatedCompletion extends Error {
  constructor(readonly bytes: number, readonly maxTokens: number) {
    super(
      "completion hit the token cap (" + maxTokens +
      ") after " + bytes + " bytes; output is incomplete",
    );
  }
}

export function parseCompletion<T>(res: ChatCompletion, maxTokens: number): T {
  const choice = res.choices[0];
  const content = choice.message.content ?? "";

  if (choice.finish_reason === "length") {
    throw new TruncatedCompletion(content.length, maxTokens);
  }
  if (choice.finish_reason === "content_filter") {
    throw new Error("completion was filtered before it finished");
  }
  if (content.trim() === "") {
    throw new Error("completion was empty (finish_reason=" + choice.finish_reason + ")");
  }
  return JSON.parse(content) as T;
}

The value of this is not that it prevents the failure — it cannot — but that it converts an unhandled SyntaxError deep in a parser into a typed error at the boundary, carrying the number that caused it. That error is actionable: raise the cap, ask for less, or split the work. An unhandled parse error, by contrast, gets caught by a generic retry somewhere and turns into three identical truncated responses and three bills.

Reproducing it without a model

There is no reason to spend a live request on this. The failure is entirely determined by two fields of the response, so construct the response.

import { describe, expect, it } from "vitest";
import { parseCompletion, TruncatedCompletion } from "../src/parse";

function completion(content: string, finish_reason: string) {
  return {
    choices: [{ index: 0, message: { role: "assistant", content }, finish_reason }],
  } as any;
}

describe("parseCompletion", () => {
  it("rejects a response cut off mid-object", () => {
    const cut = '{"items": [{"name": "apple", "note": "a red fru';
    expect(() => parseCompletion(completion(cut, "length"), 64))
      .toThrow(TruncatedCompletion);
  });

  it("rejects an empty completion", () => {
    expect(() => parseCompletion(completion("", "stop"), 64))
      .toThrow(/empty/);
  });

  it("parses a complete response", () => {
    const ok = '{"items": [{"name": "apple"}]}';
    expect(parseCompletion(completion(ok, "stop"), 64))
      .toEqual({ items: [{ name: "apple" }] });
  });
});

Keep one live test alongside these, tagged so it does not run on every commit: send a request that genuinely cannot fit — max_tokens: 16 against a prompt asking for twenty items — and assert that finish_reason comes back as length. That one asserts the provider still reports truncation the way you think it does, which is the only part of this the stubs cannot cover.

Sizing the cap so it stops happening

A guard turns a crash into an error; sizing turns the error into nothing. Estimate the output rather than guessing: take the largest instance of your schema, serialise it, and count its tokens with the same tokeniser the model uses. Add headroom of roughly a third for longer string values than you anticipated, and set the cap there.

If the honest answer is that the output is unbounded — “one object per row in the input, and the input is a file” — then no cap is correct and the design is wrong. Chunk the input, or stream and parse incrementally so that a truncated response still yields the complete objects that arrived before the cut. Streaming has its own hazards, since a partial event is not a partial object and the two boundaries do not coincide; that is the subject of parsing JSON out of a stream. What you should not do is set the cap high enough that it never fires. The cap is also your protection against a runaway generation, and removing it to fix a parse error trades a clean failure for an unbounded bill.

One retry strategy is worth building and one is not. The one that works is to retry with a larger cap and a reduced request — fewer items, shorter fields — because the second attempt is then attempting something that fits. The one that does not is asking the model to “continue” from the truncated fragment and concatenating the two responses. It looks economical and it produces documents that are subtly wrong at the join: a repeated key, a duplicated array element, a closing brace in the wrong place. If you are tempted by it, assert on the joined document rather than on the second response, and the problem shows up on the first test case.

Finally, count the failures rather than only catching them. A truncation rate is a leading indicator of outputs that are growing — a new field in the schema, a longer document being summarised, a user base writing more verbosely than the one you sized against. Emitting a metric each time TruncatedCompletion is thrown, labelled with the model and the cap, makes that visible as a slope rather than as an incident, and it costs one line in the constructor.