Embedding Drift and Stale Vectors

“Our embeddings have drifted” describes four distinct failures with four distinct fixes, and conflating them is why teams respond to a stale-vector bug by re-embedding the entire corpus. Here they are separated, each with a detector you can build this week.

Four problems with one name

Note that a model’s weights do not decay. There is no process by which a stored vector degrades on disk. Every one of these is a mismatch between two things that were supposed to correspond, which is why every detector below is a comparison rather than a measurement.

The distinction is not academic, because the four have wildly different costs to fix. Stale vectors are a bug in one code path and cost an afternoon. Corpus drift usually costs nothing at all. Query drift costs content, written by a human, over months. Provider drift costs a full re-embedding of everything. A team that has not separated them will reach for the most expensive remedy — re-embed the corpus — because it is the one that plausibly addresses all four, and will then be surprised when the symptom returns the following week because the actual cause was an ingestion path that never enqueued a re-embed.

Stale vectors: one SQL query

If you stored an embedded_at timestamp alongside the vector — and this is the argument for storing one — the detector is a single query:

SELECT count(*) FILTER (WHERE embedded_at < updated_at) AS stale,
       count(*)                                        AS total,
       max(updated_at - embedded_at)                    AS worst_lag
FROM   chunks;

Put stale on a dashboard and alert when it is non-zero for longer than your ingestion lag. That is the whole monitor, and it catches the majority of real-world “the search returns the old version of the page” reports.

A related check catches the deletion half of the problem: vectors whose document no longer exists. In a single database that is a foreign key and cannot happen. Across two systems it happens constantly, and the reconciliation job — count of ids in the vector store minus count in the source of truth — is the number to graph. A drifting difference between the two counts is the strongest available argument for keeping vectors in the same database as the rows they describe.

The provider canary

Hosted models are usually versioned, and a well-behaved provider freezes a version. But models get silently updated, endpoints get repointed, and a self-hosted deployment can be upgraded by someone who did not know it mattered. The consequence is severe: every vector you write after the change is in a different space from every vector you wrote before, with no error anywhere.

# once, at setup: store 100 fixed strings and their vectors
canary = ["the quick brown fox", "SELECT * FROM users", ...]
np.save("canary.npy", embed(canary))

# weekly, in CI:
ref = np.load("canary.npy")
now = embed(canary)
sim = (ref * now).sum(axis=1)          # both unit length

assert sim.min() > 0.9999, f"embedding model changed: min sim {sim.min()}"

Identical inputs to an identical model give identical outputs up to floating-point noise, so the threshold can be brutally tight. A similarity of 0.97 is not noise — it is a different model. This test costs a hundred embeddings a week and it is the only thing standing between you and an index quietly split into two incompatible halves.

Monitoring the two real drifts

Corpus and query drift are genuine distribution shifts and need statistics rather than assertions. Three cheap ones:

• Unanswered rate. The fraction of queries whose best similarity falls below the threshold you use to decide there is no good answer. Track it weekly. A rising line is the single most useful drift signal you can have, because it is defined in terms of the outcome rather than the mechanism.
• Query centroid movement. Average this week’s query vectors, compare with last week’s by cosine. A stable product sits high and flat; a sharp drop means your users are asking about something new, and the log of that week is worth reading directly.
• Novelty of incoming documents. For each new document, the similarity to its nearest existing neighbour. A falling average means your corpus is expanding into new territory, which is the signal to revisit chunking and to check whether the retrieval thresholds tuned on the old distribution still hold.

All three are one scalar per period, computable from data you already have, and none requires a labelled set. Store them as a time series from day one — the value is entirely in the trend, and a monitor started the week after an incident cannot tell you when it began.

What to do about each

• Stale vectors: fix the ingestion path so an update enqueues a re-embed, then backfill the affected rows. Do not re-embed the corpus; you have a bug, not a model problem.
• Corpus drift: usually nothing. A general embedding model handles new topics fine. Revisit only if the new content has genuinely different structure — much longer documents, a new language, heavy tabular content — in which case the chunker is the thing to change.
• Query drift: a content gap, not a vector problem. The queries with no good answer are a prioritised list of documents somebody should write, which is the most directly valuable artefact in this whole page.
• Provider drift: stop writing new vectors immediately, pin the model version if the provider offers pinning, and treat the recovery as a full re-embedding migration with a dual column, because that is exactly what it is.