Keeping a RAG Index Fresh: Incremental Updates

Every RAG tutorial ends at “now your documents are indexed”. The interesting half starts the following Tuesday, when someone edits a document, deletes another, and your index confidently serves both old versions.

Stale is worse than missing

A missing document produces “I don’t have information about that”, which is honest and gets reported. A stale document produces a fluent, well-cited, confidently wrong answer, which does not get reported because it looks fine. The failure is silent, and the citation makes it worse by lending it authority.

Deleted content is the sharpest version. A policy withdrawn in March that is still in your index in August will be retrieved and cited for as long as it sits there, and the only signal is a customer acting on it. This is why deletion, not insertion, is the operation that decides whether an index is trustworthy.

Chunk ids that survive a re-chunk

The naive id is doc_id + ":" + index. It is fine until a document is edited near the top, at which point every subsequent chunk shifts, every id changes, and you have rewritten the entire document’s worth of vectors for a one-word fix. Worse, any citation stored against the old ids now points at different text.

Content addressing fixes the churn:

import hashlib

def chunk_id(doc_id, text):
    h = hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
    return doc_id + ":" + h

def sync_document(doc_id, new_text, store):
    chunks = split(new_text)
    want   = {chunk_id(doc_id, c.text): c for c in chunks}
    have   = set(store.ids_for_document(doc_id))   # side index, see below

    to_add    = [c for i, c in want.items() if i not in have]
    to_delete = [i for i in have if i not in want]

    if to_add:
        store.upsert([(i, embed_one(c.text), c.metadata)
                      for i, c in want.items() if i in
                      {chunk_id(doc_id, c2.text) for c2 in to_add}])
    if to_delete:
        store.delete(to_delete)          # do this LAST, and do it always

    store.set_ids_for_document(doc_id, list(want))

The property this buys is that an edit re-embeds only the chunks whose text actually changed. Edit one sentence in a 200-chunk document and you pay for one or two embeddings, not two hundred — and every unchanged chunk keeps its id, so stored citations keep pointing at the right text.

One caveat: content-addressed ids collide across documents when the same boilerplate appears in many of them. Prefixing with doc_id as above avoids that, at the cost of storing genuinely duplicate vectors. If your corpus is heavy with repeated text, deduplicating at ingest is the better trade — and it also fixes the boilerplate-magnet retrieval problem.

The orphan problem

Look at store.ids_for_document(doc_id) in that code. Most vector stores index by vector, not by document, so “every chunk belonging to document X” is a metadata query you may or may not be able to run efficiently — and if you cannot run it, you cannot compute the delete set.

Keep a side table. One row per document holding the current chunk id set is a few kilobytes and turns deletion from a scan into a set difference:

CREATE TABLE index_manifest (
  doc_id      text PRIMARY KEY,
  chunk_ids   text[]      NOT NULL,
  source_hash text        NOT NULL,   -- skip re-chunking unchanged docs
  model       text        NOT NULL,   -- which embedding model made these
  indexed_at  timestamptz NOT NULL
);

This table also answers the questions you will eventually be asked and otherwise cannot: how many documents are indexed, when was each last touched, which ones are still on the previous embedding model, and — when someone reports a stale answer — whether that document was ever re-synced. Without it, an index is an opaque blob whose contents you can only sample.

The source_hash column is worth its own sentence. A nightly job that re-chunks and re-embeds everything unchanged is the most common way to turn a small embedding bill into a large one. Hash the source document first; if it matches, do nothing at all.

Getting the change events

The reliable architecture is a fast path plus a slow reconciliation: a webhook or CDC feed for freshness, and a nightly or weekly full listing that diffs the manifest against reality and repairs it. Every mature sync system converges on this shape, because incremental feeds always eventually miss something and only a full comparison can notice.

Changing the embedding model

Vectors from two different embedding models are not comparable. Not “less accurate” — meaningless. If half your index is embedded with model A and half with model B, the similarity scores between a query and each half are drawn from different distributions and the ranking between them is arbitrary. There is no incremental migration.

So it is a blue-green operation:

• Build a second index with the new model, from the manifest, at whatever pace is convenient. Nothing reads it yet.
• Run your evaluation set against both. This is the moment the labelled question set pays for itself — otherwise you are swapping the most consequential component in the pipeline on a vendor’s benchmark claim.
• Flip an alias. Keep the old index for a rollback window; a regression that only shows up on real traffic is common, and rebuilding takes hours you will not want to spend under pressure.
• Record the model name in the manifest and in every chunk’s metadata, so a half-migrated index is detectable by query rather than by guessing.

Dimension changes deserve a specific warning: if the new model outputs a different vector width, most stores will reject the write rather than corrupt the index — but some will accept a differently shaped collection silently if you created it fresh, and you will find out from the recall numbers rather than from an exception.

Two ordering rules make the sync safe under concurrency, and both are easy to get backwards. Write the new chunks before deleting the old ones: a reader that briefly sees both versions returns a duplicate, which is mildly annoying, whereas a reader that briefly sees neither gets told the answer does not exist. And update the manifest last, after the store has acknowledged both operations, so that a crash leaves the manifest describing a state the index has already reached rather than one it never got to. A manifest that over-claims is unrecoverable without a scan; one that under-claims is repaired by the next reconciliation.