Prompt Compression Without Losing Accuracy

Compression is usually attempted as a single heroic rewrite and evaluated by reading the output twice. Done properly it is two things: an ordering — cut the recoverable losses first — and a measurement that catches the moment you crossed a line.

Audit before you cut

Count the tokens in each block before deciding anything. A typical retrieval-augmented prompt breaks down something like this, and the shape of it usually surprises the person who wrote it:

block                        tokens    share   varies per request?
system role + rules             800     9.6%   no
few-shot examples              1500    18.0%   no
output contract                 150     1.8%   no
retrieved documents (k=8)      5200    62.4%   yes
user question                   300     3.6%   yes
tool schemas                    380     4.6%   no
                               ----
                               8330

Two facts fall out immediately. The retrieved documents are the prompt, and the part everyone rewrites first — the system rules — is under a tenth of it. Any compression effort that starts with the prose in the system prompt is optimising the wrong 10%.

Count with the tokeniser of the model you actually use rather than with a characters-divided-by-four rule of thumb. The estimate is adequate for English prose and badly wrong for the content that dominates real prompts — JSON, code, identifiers, non-English text — where it can be off by a third in either direction, which is more than enough to point the whole audit at the wrong block.

Where the money actually is

Caching changes the answer again, and this is the step almost everyone skips. The fixed blocks above (rules, examples, contract, schemas = 2,830 tokens) are a stable prefix and therefore cacheable; the documents and question are not. If your provider charges cache reads at roughly a tenth of the input rate, the effective per-call cost is:

cached prefix   2830 x 0.1  =  283 effective tokens
per-request     5500 x 1.0  = 5500 effective tokens
                               ----
                               5783

halving the system prompt  -> saves  40 effective tokens  (0.7%)
cutting k from 8 to 5      -> saves 1950 effective tokens (33.7%)

So the honest priority list is set by arithmetic rather than by aesthetics: compress what is not cached. Rewriting a cached system prompt is worth roughly a tenth of what the token count suggests, and it costs you the cache entirely for the first calls after the change.

This also reframes what compression is for. Below a certain volume it is not a cost exercise at all — it is a latency and attention exercise, because a shorter prompt prefills faster and gives the instructions less to compete with. Both of those benefits apply to cached tokens too, which the arithmetic above deliberately ignores; if the reason you are compressing is that the model keeps missing rule seven, the cache columns are not the ones to read.

The order to cut in

Ordered by how easily you can tell you went too far.

• 1 · Retrieval volume. Fewer chunks, tighter chunks, deduplicated chunks. This is the largest block and the loss is directly measurable as retrieval recall, independently of the model.
• 2 · Instructions the API now enforces. If you send a response schema, the paragraph explaining the JSON shape is dead weight. Same for stop sequences that replaced “do not add anything after the object”.
• 3 · Ceremony. “Please note that it is very important that you carefully” carries no information. Cutting it is free, and it is also usually only a few dozen tokens, so do not mistake it for the win.
• 4 · Shot count. Five to three is often flat — the in-context-learning literature has the curve saturating early. This is measurable on your eval set in an afternoon.
• 5 · Dead rules. Rules added for incidents that no longer occur. Detect them by ablation, below. This one both saves tokens and tends to improve compliance with the rules that remain.
• 6 · Documents to extractive summaries. Real savings, real risk: you lose exact wording, so quotation and citation get worse. Do not do this if the product cites sources.
• 7 · The output contract. Last, and preferably never. It is small, and it is the thing whose failure breaks the parser rather than the prose.

Two cuts that look tempting are deliberately absent. Removing the abstention instruction, because it is one line and appears to do nothing on the happy path — it does nothing until the day retrieval misses, which is the day it was written for. And shortening the examples by trimming their inputs, which changes what the examples demonstrate about realistic input length. Cut the number of examples rather than the fidelity of each one.

Automatic compression

There is a research line on compressing prompts mechanically. Jiang et al. (2023) introduced LLMLingua, which uses a small language model to identify and drop low-information tokens from a prompt, with a budget controller allocating the compression across sections; LLMLingua-2 followed with a data-distillation approach aimed at being faster and more task-agnostic. The authors report substantial compression ratios with limited degradation on their evaluation benchmarks — those are their numbers on their tasks, and the sensible way to treat them is as evidence the approach is viable rather than as a prediction about your prompt.

Three practical caveats before adopting it. The compressed prompt is not human-readable, so every future debugging session is harder. It adds a model call and its latency to every request. And it interacts badly with caching, because a compression that depends on the input produces a different prefix each time.

The harness that tells you when to stop

Compression is only safe with a metric attached. The most useful measurement is leave-one-out ablation over the rules, because it finds the dead weight and the load-bearing lines in one pass:

rules = load("prompts/triage/rules.yaml")     # each rule is one item, with an id
base  = evaluate(render(rules), cases)        # accuracy, parse rate, per-tag scores

for r in rules:
    trial = evaluate(render(rules - r), cases)
    print(r.id, trial.accuracy - base.accuracy, r.tokens)

# interpretation
#   delta >= 0            -> the rule is not paying for itself; drop it
#   delta small, negative -> keep only if tokens are cheap or the tag matters
#   delta clearly negative-> load-bearing; annotate it in the file so nobody
#                            "tidies" it away in six months

• Run at temperature zero so the deltas are not sampling noise, and remember the standard error: on 200 cases at 80% accuracy it is about 2.8 points, so treat anything smaller as no signal.
• Report parse rate separately from task accuracy. Compression usually damages format compliance first, and an average hides it.
• Keep a per-tag breakdown. A cut that costs two points overall may have removed the only instruction that handled refunds.
• Record the token count and the effective cached-adjusted count with each result, so “saved 400 tokens” can be checked against what it actually saved.

Run the ablation once properly and then leave it alone; it is not a weekly ritual. The habit worth keeping afterwards is much smaller: whenever a rule is added, record in the changelog which failure it was for. That is what gives a future ablation something to compare against, instead of a two-point delta and a guess about whether it mattered.