What a Migration Does to an Existing Prompt Compression Ratio

You have a prompt compressor in production and a number attached to it: “42% fewer input tokens”. That number was measured against one tokenizer. Migrate the model and the compressed text does not change by one byte, but the saving does — and it usually shrinks.

The claim you are carrying forward

A compression ratio is not a property of the compressor. It is a property of the pair (compressor, tokenizer). Written out, the ratio you measured was

R_old  =  tokens_old(compressed)  /  tokens_old(original)

and the number you actually want after the migration is

R_new  =  tokens_new(compressed)  /  tokens_new(original)

These are two different measurements over the same two strings. Nothing about R_old constrains R_new, because both numerator and denominator are recomputed by a different segmentation algorithm. Carrying R_old forward into a cost model after the migration is the error this page exists to stop; the rest is the arithmetic for replacing it.

Why two tokenizers disagree on one string

Tokenizers are learned vocabularies, typically byte-pair encodings, and the vocabulary is fitted to a training corpus. A string is segmented into the longest pieces the vocabulary contains, so the same string is cut differently by two vocabularies with different merge tables. This is a mechanism you can demonstrate rather than a claim you have to take on faith: run the same paragraph through two tokenizers and print the piece boundaries.

Two published facts anchor the size of the effect. Anthropic’s own documentation states plainly that OpenAI’s tiktoken is the wrong tool for counting Claude tokens, undercounting by roughly 15–20% on typical text and by more on code or non-English input, and directs callers to the /v1/messages/count_tokens endpoint instead (platform.claude.com, token counting). And within a single vendor, Anthropic’s migration guide documents that the tokenizer introduced with Claude Opus 4.7 produces roughly 1× to 1.35× as many tokens for the same content as the preceding generation, with the multiplier varying by workload shape (platform.claude.com, migration guide). Both numbers are vendor-published and both will move; treat them as evidence that the effect is first-order, not as inputs to your own model.

The arithmetic, with the assumptions labelled

Work a case with two clearly labelled assumptions, both of which you will replace with measurements from your own corpus.

Assumption A (yours to replace): the original prompt is 10,000 tokens on the old tokenizer and the compressed form is 5,800, so R_old = 0.58 — a 42% saving. Assumption B (yours to replace): the new tokenizer inflates the original prompt by a factor k_orig = 1.20 and the compressed prompt by k_comp = 1.31.

tokens_new(original)    = 10,000 × 1.20 = 12,000
tokens_new(compressed)  =  5,800 × 1.31 =  7,598

R_new = 7,598 / 12,000 = 0.633     → a 36.7% saving, not 42%

Relative change in the saving:
  saving_old = 1 - 0.58  = 0.420
  saving_new = 1 - 0.633 = 0.367
  0.367 / 0.420 = 0.874  → the compressor now delivers 87% of the
                            benefit it used to, on identical text.

Notice what did and did not change. Absolute token counts went up on both sides, so the bill went up regardless of compression. The ratio moved only because the two inflation factors differ. If k_orig and k_comp were equal the ratio would be exactly preserved and the compressor would be unaffected by the migration. The whole effect lives in the gap between them.

The asymmetry that makes the ratio shrink

Why should k_comp exceed k_orig? Because compression works by removing the redundancy that tokenizers are best at. A compressor strips filler words, contracts phrases, drops articles, replaces sentences with fragments and abbreviations. What survives is denser, less like the prose the vocabulary was fitted to, and more likely to fall outside the long merged pieces that make common English cheap. The compressed text is closer to the “code or non-English” regime where tokenizers diverge most.

So the general expectation is that a compressor loses a little effectiveness under a tokenizer change, and it loses more the more aggressive it is. A compressor that only deduplicates whole paragraphs will have k_comp very close to k_orig. A compressor that rewrites into telegraphic shorthand will not. This is testable on your own corpus in a few minutes and it tells you which of your compression stages is fragile — which is a more useful output than a single headline ratio.

Re-measuring on your own corpus

Do not apply a blanket multiplier. Both inflation factors depend on your content, and a corpus of SQL, a corpus of Japanese support tickets and a corpus of English marketing copy will give three different answers. The measurement is a loop over a representative sample.

1. Sample 200–500 real prompts from production logs, spanning the content types you actually serve. A sample of one type gives you a number that is wrong for everything else.
2. For each, keep both the pre-compression and post-compression text you already have — most compressors can be run in a dry mode that emits both.
3. Count tokens for all four sets (original and compressed, old model and new model) using each provider’s own counting endpoint. For Anthropic that is client.messages.count_tokens(model=..., messages=...); the model id is part of the request because counts are model-specific. Never estimate one vendor’s count with another vendor’s tokenizer library.
4. Compute R_old and R_new per prompt, then report the median and the interquartile range rather than the mean. Compression ratios are skewed and a mean is dominated by your longest prompts.
5. Recompute the break-even: compression costs you a preprocessing call or CPU time, and if R_new has risen enough that the saving no longer covers that cost for short prompts, add a length threshold below which compression is skipped.

Two secondary effects are worth noting because they arrive with the same change and get blamed on the compressor. Cache behaviour shifts: prompt caching has a minimum cacheable prefix expressed in tokens, and a prefix that sat comfortably above that minimum under one tokenizer can move relative to a different minimum on the new model, so a compressed prefix that used to be cacheable may stop being so — or start. And any hard token ceiling in your own code, the one that decides when to truncate or to reject an oversized request, was calibrated in old-tokenizer units; leaving it untouched means you now truncate at a different point in the text than you intend. Both are one-line fixes and neither is visible in the compression ratio itself.

The output of this procedure is a per-content-type table, not a single number, and it is the thing to store next to the compressor so the next migration has a baseline to diff against. The broader mechanics of tokenizer disagreement are covered in tokenizer comparison and token count mismatch; what is specific here is that a ratio, unlike a count, can move in a direction nobody expected because it depends on the difference between two inflation factors rather than on either one.