Conversation Compaction: Summarising Without Losing State

Every long-session assistant eventually summarises its own history to make room. Done as “summarise the conversation so far,” it reliably deletes the one sentence the rest of the session depends on, and it degrades a little more every time it runs.

What compaction is actually for

Compaction is a lossy compression of history whose loss function you get to choose. That framing is the whole of the design. The question is never “how do I shorten this?” — it is “what must survive intact, and what may be lost?” And the answer is knowable, because a conversation contains two very different kinds of content.

Most of a transcript is process: the user asked, the assistant offered three options, the user asked a clarifying question, the assistant answered, a tool ran and returned. That is safely compressible; nothing later depends on the exact words. A small minority is state: the user is deploying to eu-central-1, the budget is €400 a month, they rejected the Postgres option because of a licensing constraint, the file is called ingest.py. State is not compressible at all. Losing one item of it makes every subsequent turn wrong, and the model will not say it has forgotten — it will confidently produce an answer for us-east-1.

So a compactor that emits prose is solving the wrong problem. Prose summaries preserve process — the shape of the conversation — and discard state, because a fluent summary of a conversation naturally reads “the user asked about deployment regions and the assistant suggested several options,” which has thrown away the only fact that mattered.

The verbatim tail

Keep the most recent turns exactly as they are. Not because recent content is more important in principle, but because the immediate exchange is full of references that only resolve locally: “that,” “the second one,” “no, the other file.” Compress those and you break pronoun resolution, which produces a class of failure that looks like the model becoming stupid.

Two or three complete turns is a common floor, and it should be measured in turns rather than tokens, because half a turn is worse than none — a user message without its assistant reply invites the model to answer a question that has already been answered. Where the boundary lands is an allocation decision, and it is exactly the history block’s floor.

Compact into a record, not a paragraph

Replace the older turns with a typed structure, and make the extraction call fill in fields. The schema is doing the real work: it names, in advance, the categories of thing that are not allowed to be lost.

type SessionRecord = {
  goal: string;                  // what the user is ultimately trying to do
  constraints: string[];         // budget, region, stack, deadline, policy
  decisions: {                   // the expensive ones to re-derive
    decision: string;
    rationale: string;           // WHY, or it gets re-litigated next turn
    turn: number;                // provenance, for debugging
  }[];
  rejected: { option: string; reason: string }[];
  artifacts: {                   // by reference, never inlined
    name: string; kind: "file" | "url" | "id"; ref: string;
  }[];
  open: string[];                // unresolved questions
};

Three fields here are the ones naive summarisation always drops, and each has a specific consequence.

• rationale. A decision without its reason gets reopened. The model proposes the rejected option again at turn thirty, the user says “we discussed this,” and the session has now spent tokens re-deriving something it already paid for once.
• rejected. Negative space is information and summaries never keep it, because nothing happened. It is the highest value-per-token content in the whole record.
• artifacts by reference. A file that was produced should appear as a name and a path, not as its contents. Inlining artifacts is how a compaction step manages to increase token count, which sounds impossible until it happens — externalised working memory is the general form of the fix.

The re-summarisation trap

Here is the failure that separates a compactor that works at turn eighty from one that works at turn twenty. The obvious implementation compacts the current context, including the previous summary. So at turn 40 you summarise a summary; at turn 60 you summarise a summary of a summary. Each pass is lossy, the losses compound, and detail decays geometrically. The record gets vaguer and vaguer while remaining perfectly fluent, so nothing in your logs looks wrong.

The fix is a rule with no exceptions: always compact from the original turns, never from a previous compaction. Keep the raw transcript out of band — a database row, a file, an object store; it is not competing for window space — and regenerate the record from source each time. Compaction becomes idempotent in the sense that matters: running it at turn 60 gives the same record as running it at turn 60 would have from a fresh process, rather than a fourth-generation photocopy.

Merging is the pragmatic middle ground when re-reading everything is too expensive: extract a record from the new turns only, then merge field-wise into the existing record — append to decisions and rejected, replace goal, remove items from open that are now answered. Merging structured fields is deterministic code; merging paragraphs is another lossy model call.

The routine

async function compact(session, limits) {
  const budget = limits.historyBudget;
  if (tokens(session.turns) <= budget) return session;   // nothing to do

  // Keep whole turns from the end until the verbatim floor is used up.
  const tail = [];
  let used = 0;
  for (let i = session.turns.length - 1; i >= 0; i--) {
    const t = tokens(session.turns[i]);
    if (used + t > limits.verbatimBudget && tail.length >= 2) break;
    tail.unshift(session.turns[i]);
    used += t;
  }

  const older = session.turns.slice(0, session.turns.length - tail.length);
  if (older.length === 0) return { ...session, tail };

  // Extract from ORIGINAL turns, then merge structurally. Never re-summarise.
  const fresh  = await extractRecord(older);        // schema-constrained call
  const record = mergeRecords(session.record, fresh);

  return { ...session, record, tail, archived: session.turns };
}

Two operational notes. Compaction is itself a model call, so it costs tokens and adds latency at exactly the moment the session is already large — trigger it at a threshold below the budget (say 80% of the history allocation) so it runs before a request would otherwise fail, and run it asynchronously between turns where the interaction pattern allows. And make the record visible in your traces: when a session goes wrong at turn fifty, the first question is whether the record still contains the constraint, and that is a lookup rather than an investigation.

Finally, tell the user. An assistant that silently forgets is worse than one that says it has compacted the earlier part of the conversation, because the first teaches people to distrust everything and the second teaches them to restate what matters.