Migrating a Prompt Compression or Summarization Preprocessor

The compressor did not change when you migrated. That is the problem: it is still deciding what to delete on behalf of a model you no longer call.

There are two models in this pipeline

A prompt compressor is not a text utility. Microsoft’s LLMLingua, the most widely deployed of them, works by running a small language model — GPT-2-small or a 7B model, depending on configuration — over your context and deleting the tokens that model finds most predictable. The interface is PromptCompressor.compress_prompt(), with either a rate (keep this fraction) or a target_token (get me down to this many). Microsoft’s LLMLingua repository documents both the mechanism and the parameters.

So your pipeline contains two models with entirely separate lifecycles. The serving model is the one you migrated. The scoring model inside the compressor is the one deciding what the serving model gets to see, and your migration did not touch it. Every tuning decision you made — the rate you settled on, the fields you decided were safe to drop, the length of context you felt comfortable feeding in — was made against a pairing that no longer exists.

The same structure holds for a summarising preprocessor, where the “scoring model” is a cheap model asked to condense the document. It holds for a hand-written extractive step too, where the scoring model is a regular expression somebody wrote while looking at the old model’s failures.

The symptom is a missing field, not worse prose

Teams look for compression damage in the wrong place. They re-read a few outputs, decide the writing is still fine, and ship. The damage shows up downstream, in the parser.

Take an extraction task whose output contract is a JSON object with a policy_number field. In the source document the number is preceded by the label “Policy No.”. That label is highly predictable in context, so a perplexity-based compressor is exactly the thing that will delete it, leaving the digits floating between two unrelated paragraphs. The old serving model recovered the association from format alone. The new one does not, and returns null for that field — or, worse, returns the claim number that happens to sit two lines above.

Nothing in that failure looks like a compression problem. The prompt is unchanged, the schema is unchanged, the model is new, so the model gets the blame. The way to tell them apart takes one run: feed the uncompressed context to the new model on the failing cases. If the field comes back, the compressor is the culprit and the serving model is fine.

# The one diagnostic that separates the two causes
for case in failing_cases:
    compressed = compressor.compress_prompt(case.context, rate=0.33)
    a = extract(model="new-model", context=compressed["compressed_prompt"])
    b = extract(model="new-model", context=case.context)
    if a["policy_number"] is None and b["policy_number"] is not None:
        print(case.id, "compressor removed the evidence")

Why holding the ratio constant is the mistake

The instinct after a migration is to keep the compression rate where it was, because it is the one number everybody agreed on. But the rate was never the quantity you cared about. It was a proxy for “how much can we throw away before the answers get worse”, and the answer to that question is a property of the pairing, not of the compressor.

Two things move it. A model with a larger context window changes the reason compression exists at all — if you were compressing to fit, and you no longer need to, the honest move is to stop compressing rather than to keep a lossy step for its own sake. And a model with different attention behaviour over long inputs will tolerate a different amount of deletion in the middle of a document than at its edges. Neither is captured by a ratio.

Replace the ratio with a task metric. Sweep rate across a range on a held-out set that has ground-truth answers, plot the task metric against cost, and take the knee. That is a different number for every model, and it is the number worth carrying forward. Migrating the compression ratio itself works through that sweep in detail.

Include the compressor’s own cost in that plot, because it is routinely left out. Running a scoring model over every context is an inference pass of its own: it consumes GPU or CPU time, adds latency ahead of the first token, and has to be capacity-planned like anything else. When the serving model was expensive and the compressor cheap, that overhead vanished into the rounding. A migration that lowers the input price, or that moves you to a model with cheap cached input, can leave you paying more for the compression than the compression saves — and paying it in latency at the front of every request, where the user notices. The sweep should plot task metric against total pipeline cost and total pipeline latency, not against serving tokens alone.

Pin the structure, compress the filler

The durable fix is not a better ratio. It is to stop letting the compressor near the parts of the prompt that carry the contract.

• Never compress the instruction or the schema. LLMLingua takes instruction and question as separate arguments precisely so they can be held out. If your integration concatenates everything and compresses the blob, that is a bug independent of any migration.
• Segment the context and set per-segment rates. The structured variant, structured_compress_prompt(), exists for this: boilerplate can go to a hard rate, the section containing the fields you extract should stay near lossless.
• Force-keep the tokens that are labels. Field names, units, dates, identifiers and currency symbols are exactly the high-predictability, high-information tokens a perplexity score gets wrong. Most compressors take a list of tokens to preserve; populate it from your schema rather than by hand.
• Move the compression boundary off the few-shot examples. Compressed exemplars teach the new model the compressed style, which is a formatting change you did not intend and will see in the output.

What to re-validate, in order

Run these in sequence and stop at the first one that explains what you are seeing, so you do not tune four things at once.

1. Uncompressed baseline on the new model. Establishes the ceiling and tells you whether the compressor is implicated at all.
2. Field-level recall on the compressed path, per field. Aggregate quality scores hide the one field that vanished; per-field numbers do not.
3. The rate sweep against the task metric, on the new model only.
4. A token-budget check. If the new model prices input differently, the economic case for compressing at all may have inverted, and the cheapest correct pipeline may be the one with no preprocessor in it.
5. A re-read of what the compressor is actually deleting, on ten real documents, by eye. This is the step everyone skips and it is the one that finds the deleted label.

If compression is buying you room rather than money, compare it against plain truncation strategy, which is cheaper, has no second model in it, and fails in a way you can predict.