Migrating an Embedding Cache When You Change Models

You changed the embedding model, deployed, and retrieval got worse without a single error being logged. The cache in front of the embeddings call is the first place to look, and the reason is that it has no idea which model produced what it is holding.

Two symptoms, one loud and one silent

Changing embedding models produces one of two failures, and which one you get depends entirely on whether the two models have the same output dimension.

Different dimensions: it fails loudly. The vector store rejects the write or the query with a message naming both numbers — a dimension mismatch between the vector supplied and the index. This is annoying and it is fine. It is a build-time-shaped failure that happens at runtime, it stops immediately, and nobody serves a bad answer.

Same dimensions: it fails silently. Plenty of model pairs share an output width — successive generations from one vendor frequently do, and several vendors offer models that emit the same common sizes. When the widths agree, nothing anywhere can tell that a vector came from a different model. Every write succeeds. Every query succeeds. Every query returns exactly k results with plausible scores. Retrieval is simply worse, in a way that reads as “the model got dumber” rather than as an incident.

Why the cache serves the wrong vector space

The typical embedding cache is one line of reasoning: embedding the same string twice costs money and returns the same answer, so key on the string.

key = sha256(text.encode()).hexdigest()          # the bug
vec = cache.get(key)
if vec is None:
    vec = embed(text, model=CURRENT_MODEL)      # model is not in the key
    cache.set(key, vec)

The premise held exactly as long as CURRENT_MODEL never changed. The moment it does, the cache is a store of results from a function that no longer exists, indexed by an argument that does not identify it. Every hit returns an old-model vector; every miss returns a new-model vector. Both go into the same index and are compared with the same distance function.

What makes this specifically poisonous rather than merely wrong is that two embedding models’ output spaces are not misaligned — they are unrelated. Dimension 400 in one model has no correspondence with dimension 400 in another; the axes were arrived at independently during two separate training runs. Cosine similarity between a vector from one space and a vector from another is not a bad estimate of semantic similarity. It is a number computed from two unrelated coordinate systems, and it lands in a narrow band near zero with small variation that has nothing to do with meaning.

The consequence is specific and worth holding on to: cross-space pairs score low but not identically low, so they do not all disappear from results and they do not all stay. They compete weakly with everything, which means a document embedded in the old space is retrievable only when the correct answers are also poor. Your best queries look fine. Your marginal queries get noise. That is precisely the profile of a regression nobody can reproduce.

Confirming it in your own system

Three checks, in increasing order of effort. The first usually settles it.

1. Look at the cache hit rate across the deploy. A model swap should have collapsed it — every string is being embedded by a new function, so nothing legitimately cached applies. If the hit rate is unchanged across the cutover, the cache is serving old-model vectors and you are done diagnosing. A hit rate that stays at 90% through a model change is not good performance; it is the bug, plotted.
2. Embed a known string twice, through the cache and around it. Take any string you know was embedded before the deploy. Fetch it through the cache, then call the provider directly with the new model, and compute cosine similarity between the two vectors. For the same model on the same text this should be at or very near 1.0. A value near zero means the cached entry came from the other model.
3. Look at the score distribution in the index. If stored vectors carry a model tag — and if they do not, that is the other half of the fix — run a query set and histogram the top- k scores grouped by tag. A mixed index shows two populations: one with the spread you expect, and one clustered tightly near zero. One plot, and the conversation about whether the model “got worse” ends.

The fix is a key, not a flush

The instinct is to flush the cache. Do not start there. A flush is irreversible, it forces a full re-embed of everything at once at whatever the provider charges, it destroys the old vectors you would need to roll back, and — crucially — it does not fix anything. Flush today and the next model change reintroduces the identical bug, because the key is still wrong.

Put the model’s identity in the key. At minimum that means the provider, the model identifier, and the output dimension, since a model that supports variable output width is effectively several functions. It also has to include any request parameter that changes the result: an input-type or task-type argument (many providers embed queries and documents differently and expose that as a parameter), a truncation setting, and your own preprocessing version if you lowercase, strip or template the text before sending it.

def cache_key(text: str) -> str:
    ident = "|".join([
        PROVIDER,            # "acme"
        MODEL_ID,            # exact model string sent on the wire
        str(OUT_DIM),        # requested output dimension
        INPUT_TYPE,          # "query" / "document", if the API takes one
        PREPROC_VERSION,     # your own normalisation, versioned
    ])
    return ident + ":" + sha256(text.encode("utf-8")).hexdigest()

With the identity as a prefix rather than folded into the hash, entries for one model share a key prefix, which is what makes them enumerable and deletable later. The design of that key — what belongs in it, in what order, and what to store next to the value — is worked through in keying an embedding cache so a migration cannot poison it.

Recovering a cache that is already mixed

Deploy the new key first. That alone stops the bleeding: with the model identity in the key, every lookup for the new model misses, and the old entries become unreachable rather than wrong. No flush required, and no rollback capability lost.

The index is the part that still needs work, because it is holding vectors from both spaces with nothing distinguishing them. If your records carry a model tag you can re-embed exactly the ones with the old tag. If they do not, you cannot identify them, and the honest options are to re-embed the whole corpus or to rebuild into a new collection — the second being better, because it keeps the old collection queryable while the new one fills and makes the cutover a flag.

Whichever you choose, add the tag on the way through. Every vector should be written with the model identifier, the dimension and a run identifier in its metadata, from now on. That is what makes the next migration auditable at all — auditing whether a re-embedding actually finished is built entirely on those tags, and without them the only available completeness check is a row count, which proves nothing.