Testing That Output Length Stays Within a Product's Character Budget

A push notification gets 178 characters. A card title gets two lines before it clips. A SMS segment is 160 GSM-7 characters, or 70 if any character is not in that alphabet. None of those budgets is expressible as max_tokens, and setting max_tokens to approximate one produces a mid-word cut instead of a short answer.

max_tokens is not a character limit

The conversion is not a constant. English prose runs somewhere near four characters per token, but the ratio collapses toward one for scripts without a Latin alphabet, for code, for long digit strings and for unusual proper nouns. A budget derived by dividing your character limit by four is a budget that is roughly right for English and wildly wrong for Japanese, which is precisely the market where a clipped notification looks worst.

The other half of the problem is what max_tokens does when it binds. It does not shorten the answer; it stops generation, mid-word and mid-sentence, and reports a truncation via the finish reason. You asked for a short answer and received a long answer with the end removed. So max_tokens is a cost and runaway guard, and it belongs in the request, but it is not the mechanism that enforces a character budget.

The mechanism is three things at once: ask for the budget in the prompt, assert it on the response, and enforce it deterministically before rendering. Each of those needs a different test.

Counting characters is harder than it looks

Whatever counts characters in your test must count them the same way the thing enforcing the budget does, and in JavaScript the default does not. String.prototype.length returns UTF-16 code units: a single emoji outside the basic multilingual plane counts as two, a flag emoji counts as four, and an emoji with a skin-tone modifier and a zero-width joiner counts as many more. A name with a combining accent counts as two where a user sees one.

If your budget exists because a UI clips, the unit that matters is the user-perceived character — the grapheme cluster. Count those with Intl.Segmenter, which is in the platform and needs no dependency.

const graphemes = new Intl.Segmenter(undefined, { granularity: "grapheme" });
export const graphemeLength = (s: string) => [...graphemes.segment(s)].length;

// "👍🏽" — 4 UTF-16 code units, 2 code points, 1 grapheme
expect("👍🏽".length).toBe(4);
expect(graphemeLength("👍🏽")).toBe(1);

Where the budget comes from a downstream system rather than from pixels, use that system’s unit instead and say so in the test name. An SMS gateway counts GSM-7 septets and falls back to UCS-2 for the whole message if one character is outside the alphabet, so a single curly apostrophe in an otherwise plain message more than halves the available length. That is a rule about the gateway, not about the model, and it belongs in a named function with its own unit tests.

Three layers, three tests

1. The prompt asks for the budget, in the right unit. Test that the rendered prompt actually contains the constraint and the current number — a template where the budget was hard-coded in prose and the config later changed is a real and boring bug. State the limit as characters and give the model a target below the hard cap, because models approximate length rather than count it.
2. The response is asserted against the budget. Run the real model over a fixture set of inputs chosen to provoke length — the longest realistic input, the most complex request, one in each supported language — and assert the grapheme count. Sample each case several times and assert on the maximum observed, not the mean; the budget is violated by the worst response, not the average one.
3. The enforcement path is deterministic. The assertion in step two will fail sometimes, because a model cannot be relied upon to count. So there is a truncator, and it is what actually protects the UI. Test it independently of the model.

Keeping these separate is what stops the suite from being flaky. Step two is a quality signal about the prompt and belongs on a schedule with a tolerance for occasional failure; step three is a hard invariant and belongs on every commit, where it will never flake because it involves no model at all.

The truncator needs its own suite

A truncator that slices at index n will cut a grapheme cluster in half and emit a lone surrogate, which renders as a replacement character and, in some pipelines, fails JSON serialisation downstream. It will also cut mid-word, which looks like a bug even when it is policy. The cases:

• Input already under budget is returned unchanged — identity, not a copy with a trailing ellipsis.
• Input exactly at budget is returned unchanged. Off-by-one lives here.
• Input one over budget produces output at or under budget including the ellipsis, if you append one. The ellipsis counts.
• A cut that would land inside a grapheme cluster moves to the boundary before it, never after.
• A cut that would land mid-word moves back to the previous word boundary, for languages that have them — and does not, for languages that do not space words, where Intl.Segmenter with word granularity is the right tool rather than splitting on spaces.
• Input that is entirely one long token — a URL, a hash — still gets truncated rather than returned whole because no word boundary was found.
• Empty input, whitespace-only input, and a budget of zero all return without raising.

it.each([
  ["under budget", "hello", 20, "hello"],
  ["exactly at budget", "hello", 5, "hello"],
  ["one over, cuts to word boundary", "hello there", 8, "hello…"],
  ["never splits a grapheme", "ok 👍🏽👍🏽", 4, "ok 👍🏽"],
  ["no word boundary available", "https://example.com/very/long", 10, "https://e…"],
])("%s", (_name, input, budget, expected) => {
  const out = truncate(input, budget);
  expect(out).toBe(expected);
  expect(graphemeLength(out)).toBeLessThanOrEqual(budget);
});

The final assertion in that block is the important one and it holds for every row regardless of the expected string, which makes it a property worth asserting separately over generated input as well.

Setting a budget you can defend

Derive the number from the constraint rather than choosing a round one. If a card clips at two lines, measure the characters that fit at your smallest supported viewport in your widest supported script, and set the hard cap there. Then set the prompt’s stated target meaningfully below it — enough headroom that ordinary variation does not reach the truncator, since every truncation is a sentence a user reads half of.

Record both numbers in one place and have the prompt template, the assertion and the truncator all read from it. Three copies of a budget diverge, and the failure is silent: the prompt asks for 160, the test asserts 180, the truncator cuts at 200, and the UI clips at 150.