Re-Embedding: What Happens When You Change Models

A better embedding model ships. Your index has 40 million vectors from the old one. The vectors are not upgradeable, not convertible and not mixable, and the migration is entirely an exercise in never having both kinds answer the same query.

Why the old vectors are worthless

Each model learns its own coordinate system. Dimension 412 means something to model A and something unrelated to model B; there is no rotation you can apply, because nothing constrains the two spaces to be related at all. Even the same model at a different version has this property unless the vendor explicitly promises otherwise.

The failure mode when they mix is not an error. It is a similarity score computed between two vectors from different spaces, which is a real number, plausibly in range, and meaningless. In a mixed index the new documents systematically outrank the old ones or vice versa, and the only symptom is that search quality is worse for reasons nobody can reproduce. This is why the migration is worth planning rather than improvising.

The one column that makes this safe

ALTER TABLE chunks
  ADD COLUMN embedding_v2      vector(1536),
  ADD COLUMN embedding_model   text NOT NULL DEFAULT 'text-embedding-3-small',
  ADD COLUMN embedded_at       timestamptz;

CREATE INDEX CONCURRENTLY chunks_v2_hnsw
  ON chunks USING hnsw (embedding_v2 vector_cosine_ops)
  WHERE embedding_v2 IS NOT NULL;

A second column rather than an in-place update, because in-place means the index is half one model and half the other for the whole duration of the backfill — precisely the state that produces silently wrong results. With two columns the old one keeps serving, correctly and entirely, until you switch.

The embedding_model column is the cheap insurance. It makes “which model produced this vector” a query rather than an archaeology exercise, it makes a partial backfill visible, and it turns the next migration from a research project into a filtered update. If you take one thing from this page and you are not migrating today, add that column.

The partial index on embedding_v2 IS NOT NULL matters too: it lets the index grow with the backfill rather than being built at the end, and CONCURRENTLY keeps writes flowing while it does.

The migration, in seven steps

• 1. Evaluate before you commit. Embed a 1% sample with the new model and score it against your gold query set. A migration you cannot justify with a number is a migration you should not start — models are not strictly ordered, and a newer one can be worse on your domain.
• 2. Add the column and the partial index. As above. No behaviour change; nothing reads the new column yet.
• 3. Dual-write new documents. From this moment everything ingested gets both vectors. This bounds the backfill: it has a fixed end rather than chasing a moving corpus.
• 4. Backfill oldest-first, idempotently. Select where embedding_v2 IS NULL, batch, write, commit. Crash-safe by construction — restart and it resumes.
• 5. Shadow-read. Run both retrievals for a slice of live traffic, serve the old results, and log the overlap between the two top-10 lists. Low overlap is expected and is not itself bad; what you are watching for is the new one returning nothing, or returning the same handful of documents for everything, both of which indicate a pipeline bug rather than a model difference.
• 6. Cut over behind a flag, percentage by percentage, with the old column still populated. Rollback is a flag flip, which is the entire reason for the dual-column shape.
• 7. Drop the old column after a fortnight, not on cut-over day. Reclaiming the space is worth far less than being able to go back.

Backfill arithmetic

The timeline is almost never bounded by your workers. Take 20 million chunks at 400 tokens:

total tokens   = 20e6 * 400            = 8,000,000,000
batching       = 256 inputs per request
requests       = 20e6 / 256            = 78,125

bounded by concurrency:
  78,125 / 8 workers * 0.5 s/request   =  4,883 s  =  1.4 hours

bounded by a 5M tokens-per-minute limit:
  8e9 / 5e6                            =  1,600 min = 26.7 hours

-> the rate limit binds, by nearly 20x. Adding workers does nothing.

That is the number to compute before you promise anyone a date. The consequences follow from it: request a limit increase rather than scaling workers, run the backfill at a deliberately throttled rate so it does not starve your live traffic of the same quota, and use the largest batch size the API allows because per-request overhead is pure waste at this volume. Check the provider’s maximum inputs per request and its maximum total tokens per request — batching too aggressively earns a rejection rather than a partial result.

Add the money, which by comparison is usually the easy part: 8 billion tokens at a rate of $0.02 per million is $160; at $0.13 per million it is $1,040. Both are smaller than the engineering time, which is the usual finding and worth stating explicitly so nobody optimises the wrong term.

Verifying you did not regress

Three checks, in increasing order of how much they tell you. Gold-set recall@10 before and after is the fast one and it runs in minutes. Top-10 overlap on shadow traffic tells you how much has changed, which is context for interpreting everything else. And a click-through or answer-quality comparison over a week of real traffic is the only one that measures whether users are better off — start it on cut-over day rather than deciding you need it later.

Also verify a boring thing: that every row has a vector and that the embedding_model column has exactly one distinct value when you are done. SELECT embedding_model, count(*) FROM chunks GROUP BY 1 is the whole check, and a straggler group of forty thousand rows is a very common outcome of a backfill that ran alongside an ingestion pipeline nobody remembered to update.