Context Compression: Making 100k Tokens Fit in 10k

Compression is the one context technique that costs money to apply, so it is the one that most needs an arithmetic justification. Two cheaper things usually beat it, and knowing when they do not is most of the skill.

The free wins, first

Before any model-based technique, remove the tokens that carry no information. This is unglamorous, lossless, and routinely worth 30–60% on real prompts:

• Strip markup. Scraped HTML in a prompt is mostly attributes, navigation and script tags. Extract text.
• Minify structured data. Pretty-printed JSON pays for every space. For tabular data, CSV or a compact key-per-column form is dramatically cheaper than an array of objects that repeats every key on every row.
• Deduplicate. Retrieval with overlapping chunks delivers near-identical passages. Hash and drop.
• Drop what cannot be read. Base64 blobs, long IDs, hashes, minified bundles. They are incompressible by the tokenizer and useless to the model.
• Trim tool descriptions. Every word in a JSON Schema description is billed on every call in the loop.

Four families that cost something

Compression has to pay for itself

Every model-based technique spends tokens to save tokens. The arithmetic, with illustrative rates you should replace:

original context T  = 20,000 tok      target model input   $3.00 / M
compressed T'       =  4,000 tok      compressor in/out    $0.15 / $0.60 per M

saving per use    (20,000 - 4,000) / 1e6 * $3.00       = $0.0480
compression cost  20,000/1e6*$0.15 + 4,000/1e6*$0.60   = $0.0054

net per use, if the compressed artefact is reused      = +$0.0426
net on a single use (compress then immediately send)   = +$0.0426 as well

BUT compare against caching the same 20,000 tokens:
  cache read at a 0.1x multiplier: 20,000/1e6 * $0.30  = $0.0060
  compressed prompt, uncompressed rate: 4,000/1e6*$3   = $0.0120

Read the last two lines carefully, because they are the finding that matters. When the context is stable, caching is both cheaper than compression and lossless — there is no reason to compress a static system prompt. Compression earns its place when the context is different on every request, when it is far too large to cache economically, or when it must be reused across models that do not share a cache.

The full decision order is therefore: delete what is useless, retrieve rather than stuff, cache what is stable, and compress what is left. Most teams reach for the fourth first.

Latency deserves its own line in the arithmetic, because compression moves cost around rather than only downward. A compression call is an extra network round trip and an extra generation, both on the critical path, and the compressor has to read the whole context before it emits anything. For an interactive feature that can easily add more wall-clock time than the shorter prompt saves. Compression is therefore most defensible where it can happen offline — at ingestion, or on a schedule — and least defensible in the middle of a request a human is waiting on.

What each family destroys

Compression is lossy by definition; the question is which loss you can tolerate. Extractive selection loses the connective tissue between passages, so it damages tasks requiring synthesis across them. Abstractive summarisation loses precision — exact figures, dates, names and, notoriously, negations, since “the clause does not apply to subsidiaries” and “the clause applies to subsidiaries” summarise to nearly the same embedding-space neighbourhood. Token pruning loses grammatical scaffolding, which is usually recoverable by the model and occasionally not, and it makes your prompts unreadable to the human debugging them at 2am.

One loss is shared by all of them and is worth stating plainly: after compression you can no longer quote the source. If your product shows citations, the compressed text must retain identifiers back to the original passages, or the citation feature quietly starts fabricating.

There is also a compounding effect to watch for in long-running sessions. Compressing a context that already contains a summary of an earlier compression is a summary of a summary, and the loss is not additive but multiplicative — specifics disappear first, then qualifiers, then the distinction between what was established and what was assumed. Keep a pointer back to the uncompressed original at every level, and re-compress from the source rather than from the previous compression whenever you can afford to.

Rules that prevent the worst outcomes

• Never compress the instruction. The system prompt and the user’s question go through untouched. They are small and they are the specification.
• Never compress structured values. Identifiers, amounts, dates, quantities. Extract them into a small verbatim block and compress only the prose around them.
• Compress once, cache the result. If the same source document is used repeatedly, the compressed form is a stable artefact — store it rather than regenerating it per request.
• Measure end to end, not on compression ratio. A 10× ratio that costs three points of answer accuracy is a bad trade at almost any price, and you will not see it unless the evaluation runs on final answers.
• Keep the uncompressed path. When an answer is wrong, the first diagnostic is to rerun it with the full context. If that fixes it, compression is your bug; if not, it never was.