Testing That Long Inputs Are Truncated Consistently

Truncation bugs do not throw. They produce a prompt that is slightly wrong, a model that answers slightly oddly, and a debugging session that starts with the model and takes a day to reach the string handling.

Test it away from the model

Truncation is the easiest thing in an LLM application to test properly, and it is usually tested worst, because it is written inline in the function that builds the prompt. Pull it out. A function from a message list and a token budget to a message list has no I/O, runs in microseconds, and admits assertions that are exact rather than statistical — a rarity on this ground.

The counting function goes with it. If truncation uses a real tokeniser, inject it, so the tests can use both the real one and a trivial fake where one character is one token. The fake makes boundary cases readable: a budget of 10 and an input of eleven letters is a case anyone can check by eye, and the real tokeniser then gets a smaller set of tests about its own behaviour.

The three inputs that matter

Off-by-one errors here are common because there are two plausible meanings of a budget — at most N, or fewer than N — and the code and the caller frequently disagree. Test the boundary explicitly, in all three positions.

import { describe, it, expect } from "vitest";
import { truncate } from "./truncate";

const count = (s: string) => s.length;   // fake tokeniser: 1 char = 1 token

describe("truncate at the boundary", () => {
  it.each([
    ["under budget", "abcdefghi", 10, "abcdefghi"],
    ["exactly at budget", "abcdefghij", 10, "abcdefghij"],
    ["one over budget", "abcdefghijk", 10, "abcdefghij"],
  ])("%s", (_n, input, budget, expected) => {
    expect(truncate(input, budget, count)).toBe(expected);
  });

  it("never exceeds the budget for any input", () => {
    for (const len of [0, 1, 9, 10, 11, 100, 10_000]) {
      expect(count(truncate("x".repeat(len), 10, count))).toBeLessThanOrEqual(10);
    }
  });
});

The empty-input case belongs in that list too. A truncator that returns undefined for an empty string is a crash in a code path that only fires when a user submits a blank message, which they do.

There is a fourth boundary that is easy to miss and expensive to get wrong: the budget you pass in is rarely the model’s context window. It is the window minus the reserved output allowance, minus the tool schemas, minus whatever the provider adds around your messages for formatting. Each of those subtractions is a place to be off by a few hundred tokens, and being off in the optimistic direction produces a context-length error from the provider on precisely the longest and most valuable requests. Assert the reservation arithmetic separately from the slicing: given a stated window, a stated output reservation and a stated schema size, the budget handed to the truncator is the remainder and is never negative. That test costs one line and removes a category of production error that is otherwise diagnosed by reading a stack trace at the wrong layer.

Four invariants

Length is the least interesting property. These four are where the real bugs live.

• Bound. The result is always within budget, for every input including inputs already under it. Stated as a universally quantified property, not as three examples.
• Idempotence. Truncating twice equals truncating once. A truncator that appends an ellipsis or a marker usually fails this, because the second pass truncates the marker too and the result shrinks each time — which is how a retried request ends up shorter than the original.
• Prefix stability. Adding text to the end of an already-over-budget input must not change the output. If it does, the truncator is not taking a prefix, and two requests that should share a cache key will not.
• Well-formedness. The output is valid text. Cutting at a byte offset can split a multi-byte character or a surrogate pair, producing a replacement character or an encoding error two layers away. Assert the output round-trips through your encoder, with a fixture containing emoji and combining characters.

Idempotence and prefix stability are the two that catch the majority of real defects, and neither is checked by any test that only looks at length.

Property tests find the rest

Those invariants are stated as universals, so state them that way in code. Property-based testing libraries generate the inputs — fast-check in the JavaScript ecosystem, Hypothesis in Python — and shrink a failure to a minimal example, which for string handling is worth a great deal: the reported counterexample tends to be a two-character string that shows you the bug directly.

import fc from "fast-check";

it("is idempotent and bounded for any input and budget", () => {
  fc.assert(
    fc.property(fc.string(), fc.integer({ min: 0, max: 200 }), (s, budget) => {
      const once = truncate(s, budget, count);
      expect(count(once)).toBeLessThanOrEqual(budget);
      expect(truncate(once, budget, count)).toBe(once);
    }),
  );
});

Generate the budget as well as the input. A budget of zero is a real case — it happens when the system prompt alone fills the window — and code that assumes a positive budget produces a negative slice index, which in JavaScript quietly returns the wrong end of the string. That is the bug a property test finds in the first hundred cases and a hand-written suite never contains.

What must never be dropped

A budget-respecting truncator can still be wrong, because not all of the prompt is equally droppable. Truncation policy is a product decision, and each part of it is a test.

1. The system prompt survives intact. Assert that for every budget above the system prompt’s own size, the output contains it verbatim. A truncator that trims the system prompt changes the model’s instructions under load, which produces behaviour that varies with conversation length.
2. The last user message survives. Dropping the question to keep the history is the most confusing possible failure: the model answers an earlier turn.
3. Tool definitions survive, or the tool loop is disabled. Half a tool schema is worse than none. Assert that the schema block is either entirely present or entirely absent, never sliced.
4. Messages are dropped whole. Assert the output messages are a subsequence of the input messages with contents unmodified, except at most one message explicitly marked as trimmed. A truncator that cuts a message mid-sentence and leaves it unmarked hands the model a false premise.
5. Something records that it happened. Assert the function returns how many messages and tokens it dropped, and that the caller emits it. Truncation you cannot see in a trace is the reason for a whole genre of unreproducible bug report; the general treatment is in prompt truncation.