Why You Have to Re-Embed When You Change Embedding Models

Yes, all of it, and the reason is worth understanding rather than taking on faith — because the same reason tells you the order to do it in, and the order is the part people get wrong.

What a vector actually is

An embedding model is a learned function from text to a point in a space of some fixed dimension. The dimension is a published number. The axes are not published, because they are not anything: they are an artefact of one training run, of that run’s data, objective, initialisation and random seed. There is no dictionary anywhere saying that dimension 411 means “legal register” or that dimension 12 means “past tense”. The coordinates only mean something relative to the other coordinates produced by the same model.

That is the whole of it. Retrieval works because the model was trained so that texts a reader would call similar land near each other under some distance measure — usually cosine, sometimes inner product, occasionally L2. The guarantee is about pairs of points from the same model. Nothing in the training of model A said anything about where model B puts things, because model B did not exist to A and never will. Two independently trained models produce two unrelated coordinate systems that happen to have the same number of axes.

People reach for the intuition that the two spaces must be “roughly aligned” because both models were trained on broadly similar text and both learned broadly similar notions of meaning. It is a reasonable intuition and it is wrong in the way that matters. Even if the two spaces encoded exactly the same information, they would encode it under an arbitrary rotation and rescaling of each other, and cosine similarity is not invariant to comparing across two different rotations. Aligning two embedding spaces is a real research technique — you fit a transformation on a set of texts embedded by both models — but it needs that paired data, it is lossy, and it is strictly more work than re-embedding.

A worked example in three dimensions

Take three documents and two models, and shrink the space to three dimensions so the arithmetic fits on a line. Model A places them like this:

A("cat")     = (0.80, 0.60, 0.00)
A("dog")     = (0.70, 0.70, 0.10)
A("invoice") = (0.00, 0.10, 0.99)

Within model A this behaves exactly as you want. The cosine between cat and dog is high; the cosine between cat and invoice is near zero. Now model B, trained separately, on the same three strings:

B("cat")     = (0.10, 0.99, 0.05)
B("dog")     = (0.15, 0.98, 0.02)
B("invoice") = (0.90, 0.10, 0.40)

Within model B this also behaves exactly as you want: cat and dog are almost on top of each other, invoice is far from both. Both models are good. Both are internally consistent. Now suppose your backfill is half done, so the query is embedded with A and the index contains a mixture. Compare the A-space query for cat against the B-space documents:

cos( A(cat), B(cat) )
  = (0.80·0.10 + 0.60·0.99 + 0.00·0.05) / (1.000 × 0.996)
  = 0.674 / 0.996 = 0.677

cos( A(cat), B(invoice) )
  = (0.80·0.90 + 0.60·0.10 + 0.00·0.40) / (1.000 × 0.990)
  = 0.780 / 0.990 = 0.788

The invoice scores higher than the cat. A query about cats retrieves the invoice first, with a similarity of 0.79, which is a number that looks like a good match and would clear most thresholds people set. This is not noise and it is not a near miss; the ranking is inverted, and it is inverted confidently. Scale that from three dimensions to fifteen hundred and from three documents to three million and you have a retrieval system that returns plausible, well-scored, unrelated material for an unpredictable fraction of queries.

Why nothing raises an error

Cosine similarity is defined for any two vectors of the same length. If both models output 1,536 dimensions, every layer of your stack is satisfied. The database accepts the insert, the index accepts the vector, the distance operator returns a float between −1 and 1, the top-k comes back with the right number of rows and the scores are in a normal-looking range. There is no NaN, no exception, no warning line in a log. The only symptom is that the answers get worse, and “the answers got worse” is the hardest class of bug to attribute, because it arrives at the same time as a dozen other changes and it is reported by users rather than by monitoring.

This is the reverse of the case where the dimensions differ, which is covered in the dimension-mismatch error a vector database throws on insert. That error is the lucky outcome: it stops you at the first row. The dangerous migration is the one between two models that happen to share a width, because it will run to completion and report success.

There is a second version of the same bug that does not involve a backfill at all, and it is more common than the first: shipping a new embedding model on the query path while leaving the documents alone. The query encoder and the document encoder must be the same model. Because the change to the query path is one line and needs no migration, it is easy to deploy on its own, and it produces exactly the failure computed above across the entire corpus at once.

The cutover is per index, not per document

The operational consequence follows directly. If you re-embed in place — iterate over the corpus, call the new model, overwrite the vector column — then for the whole duration of the backfill every query is comparing a query vector against a mixture of two spaces. Retrieval is not degraded during that window, it is broken during that window, and the breakage is worst in the middle. A backfill that takes eighteen hours is eighteen hours of a production system returning arbitrary results.

So the unit of atomicity is the index, not the row. Build the new embeddings into a separate index, table, collection or namespace while the old one continues to serve. Flip the read path once, when the new index is complete. Every reader is either entirely in the old space or entirely in the new one, and there is no moment at which a query straddles both. That is also, conveniently, what makes a rollback possible at all — see keeping the old index live and dual-writing during the backfill, which is the same shape of plan for a different reason.

Two per-index rules fall out of this and are worth stating explicitly, because both get broken by well-intentioned incremental deploys. First, the similarity threshold does not travel. Different models produce different score distributions, so a cutoff of 0.75 tuned on the old index will either flood or starve the new one, and the resulting change in result counts is easy to misread as a quality collapse. Recalibrate the threshold on the new index against the same queries. Second, any index parameter tuned to the old vectors — the number of IVF lists, an HNSW ef_search, a quantisation configuration — was tuned to a distribution that no longer exists.

What survives, and the one column that does not

Re-embedding is less total than it sounds, and knowing exactly what is invalidated keeps the work bounded. What survives untouched: the source documents, your chunk boundaries and chunk text, document and chunk identifiers, all metadata you filter on, your ingestion pipeline, and the mapping from chunk back to source for citations. If you are not also changing your chunking strategy, do not change it in the same migration — two variables moving at once means you cannot attribute the result, and chunking decisions deserve their own evaluation.

What is invalidated is the vector column and everything computed from it. That second clause is the one that is under-counted: cluster assignments, deduplication decisions made by a similarity threshold, precomputed “related documents” tables, semantic cache keys, and any fine-tuned adapter or reranker trained on top of the old base model. The bill for all of it is worked through in estimating the cost of re-embedding everything downstream.

Finally, one genuine partial exception, so it does not read as a loose end. Some model families are trained so that a longer vector can be truncated to a shorter one with graceful rather than catastrophic quality loss — the shorter vector is a prefix of the longer one. Within such a family, a 1,536-dimension output and the first 1,536 dimensions of a 3,072-dimension output from the same model are related in a way that two unrelated models never are. That is a real property and it is useful for storage, but it is not an escape from this page: it is one model, and a truncated vector still has to be renormalised and still cannot be compared against a differently-truncated one. Across two models, there is no exception at all.