Truncation Strategies When Your Prompt Won’t Fit

Sooner or later the conversation is longer than the window. Every option loses something; the engineering decision is choosing what to lose, deliberately, rather than finding out from a user that the assistant forgot its instructions.

The invariants first

Before ranking strategies, fix the things no strategy is allowed to break. Violating any of these turns a quality degradation into an outage:

• The system prompt survives intact. It is the only part of the request that defines behaviour. Dropping it to save tokens produces an assistant that is not your product.
• The final user message survives intact. Truncating the actual question to fit the history is exactly backwards.
• Tool calls and their results stay paired. An assistant turn containing a tool call must be followed by the matching result, and vice versa. This is the invariant that FIFO truncation breaks first and most often.
• Messages stay whole. Cutting a message in half leaves dangling markup, half a JSON object, or half a code block, and the model will faithfully continue the mess.
• Few-shot examples stay complete. Half an exemplar is worse than no exemplar — it teaches the wrong shape.

Five strategies, worst to best

The ranking is by information preserved per token, not by effort. In practice most products should be at 3, move to 4 when sessions run long, and reach 5 only when the history is genuinely a corpus.

One decision sits above the choice of strategy: where truncation happens. If you let the provider do it — and some endpoints will silently drop the oldest messages for you — you have outsourced a product decision to a default you did not read, and you will not be told which turns went. Doing it yourself costs a function and buys you the ability to log what was dropped, to tell the user, and to choose the strategy per feature rather than per vendor.

Make it observable while you are there. Two counters — turns dropped and tokens dropped, per request — turn “the assistant keeps forgetting things” from a vague complaint into a query. A steady rise in either is usually a prompt that grew, not a user whose conversations got longer.

The errors bad truncation produces

Orphaning a tool call is the characteristic failure and both major API shapes reject it explicitly rather than degrading. The OpenAI-shaped message is a 400 along the lines of:

An assistant message with 'tool_calls' must be followed by tool
messages responding to each 'tool_call_id'.

Anthropic’s complains in the other direction, about a tool_result whose tool_use_id has no matching tool_use block in the preceding message. Both mean the same thing: your truncator treated the message list as a flat array when it is really a list of transactions, some of which span two entries.

Treat the pair as one unit. If you cannot fit both, drop both — and if the tool result is enormous, truncate the content of the result with an explicit marker rather than removing the message:

{"role": "tool", "tool_call_id": "call_9f2",
 "content": "…first 400 tokens…\n\n[truncated: 18,400 tokens omitted]"}

The marker matters. A model given a silently shortened result will reason as if it saw everything; one told the result was truncated will often ask for a narrower query instead.

A budgeted packer

The whole strategy collapses into one function that fills a budget from the ends inward, respecting the invariants:

def pack(system, history, question, budget, count):
    """history: list of turn-groups; a tool call and its result are ONE group."""
    fixed = count([system]) + count([question])
    room = budget - fixed
    if room < 0:
        raise PromptTooLong("system + question alone exceed the budget")

    kept_head, kept_tail = [], []
    head, tail = 0, len(history) - 1
    take_from_head = True                      # alternate: preserve framing AND recency

    while head <= tail:
        i = head if take_from_head else tail
        cost = count(history[i])
        if cost > room:
            break
        room -= cost
        (kept_head if take_from_head else kept_tail).append(history[i])
        head, tail = (head + 1, tail) if take_from_head else (head, tail - 1)
        take_from_head = not take_from_head

    dropped = tail - head + 1
    middle = ([{"role": "system",
                "content": f"[{dropped} earlier turns omitted]"}]
              if dropped > 0 else [])
    return [system, *kept_head, *middle, *reversed(kept_tail), question]

Two details are load-bearing. The alternation means you keep framing and recency rather than choosing between them. And the omission notice is not decoration: without it the model has no way to know the conversation is incomplete, and will confidently answer questions about material it was never shown.

Finally, decide what the user is told. Silently forgetting is the worst option available: it is indistinguishable from the assistant being careless, and it destroys trust faster than an honest limit would. A one-line notice in the interface — that earlier parts of the conversation are no longer being considered, with an offer to start a fresh thread — converts a mysterious quality regression into a comprehensible constraint. Where the content itself is compressible rather than droppable, context compression is the better tool.