Incremental Processing: Only Redo What Changed

Every pipeline is incremental eventually. The question is whether it was designed that way or retrofitted after the first time somebody changed a chunk size and the answer was “that will take eleven hours and cost the embedding budget again”.

The bill a full rebuild pays

Put numbers on it before designing anything, with every input labelled as an assumption you will replace.

Assume a corpus of D = 200,000 documents averaging T = 4,000 tokens, so 800M tokens. Assume an embedding price of p per million tokens. A full re-embed costs 800 × p. Assume further that per week 1% of documents are new and 2% are modified — that is 3% of 800M, or 24M tokens, costing 24 × p.

The ratio is 33×, and it is the entire argument. Note also what it is not sensitive to: the absolute price. Whatever p turns out to be, and however it changes, incremental costs about 3% of full for this change rate. The break-even is a change rate of 100%, which you will never reach. The only regime where full rebuilds win is when D is small enough that the engineering is not worth it — and that threshold is about an hour of wall-clock, not a dollar figure.

Wall-clock has the same shape and is often the binding constraint: an eleven-hour rebuild means you get one experiment per day, which means you will not run the chunking experiment at all.

Why mtime is not the signal

The obvious change detector is a modification timestamp. It is wrong often enough to be dangerous, in both directions.

• False positives. A file copy, an rsync without --times, a checkout, a restore from backup, or a nightly export that rewrites every row — all bump mtime with identical content. Each one triggers a full reprocess of everything.
• False negatives. Some systems preserve mtime across edits, and clock skew between machines can move it backwards. “Newer than the last run” then silently skips real changes, which is the worse failure because nothing is logged.
• No granularity. mtime tells you the file changed. It does not tell you whether the change was to the article text or to the copyright year in the footer — and only one of those is worth an embedding call.

Content hashes have none of these properties. Use mtime and size as a cheap pre-filter to decide which files to open — this is what rsync and git’s stat cache do — and the hash as the decision. And hash the normalised extracted text, not just the raw bytes: a PDF re-exported from the same source has different bytes and identical text, and it is the text you were going to embed.

A cache key per stage

The key for a stage’s output must include everything that could change the output: the input’s hash, the stage’s code version, and any configuration the stage reads. Miss one and you get stale artefacts that survive a deploy, which is the hardest class of bug in this whole area because the pipeline reports success.

import hashlib, json

def stage_key(stage: str, code_version: str, config: dict,
              *input_hashes: str) -> str:
    payload = json.dumps({
        "stage": stage,
        "code": code_version,          # bump when behaviour changes
        "config": config,              # chunk_size, model id, thresholds...
        "inputs": sorted(input_hashes),
    }, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.blake2b(payload, digest_size=16).hexdigest()

# chunker config is part of the key, so changing chunk_size invalidates
# chunking and embedding, and nothing else.
key = stage_key("chunk", "7", {"size": 800, "overlap": 100}, text_hash)

Putting config in the key is what makes experiments cheap: run the pipeline with size=600, get a complete second set of chunks and embeddings alongside the first, compare them, and throw away the loser by deleting one prefix. Nothing was overwritten, so there is nothing to restore.

Sort the input hashes and serialise the config canonically. A key that depends on dictionary ordering produces cache misses that look like random reprocessing, and you will spend a day on it.

Two things must stay out of the key. Timestamps, obviously — a key containing the current time is a key that never hits. Less obviously, anything that varies per worker or per environment: a hostname, a temporary path, a request id that found its way into a config object. The rule that catches all of them is that the key must be computable twice on different machines and come out the same, so write a test that asserts exactly that on a fixed input, and run it in CI. It is four lines and it prevents the entire class.

What a version bump should invalidate

Because each stage keys on its inputs, invalidation cascades on its own — but only downstream, which is exactly right:

The third row is the one that saves a weekend. Migrating to a new embedding model is unavoidable eventually, and when chunks are content-addressed it is a read of existing artefacts and a write of new vectors, with the old index still serving traffic until the new one is complete.

Deletions and the orphan problem

Incremental pipelines are good at additions and changes, and they miss deletions structurally: nothing arrives to trigger work. The source document is simply gone, and the chunks and vectors derived from it stay, retrievable, forever.

The fix is a reconciliation pass on a schedule — set difference between the source’s current id list and yours:

def reconcile(source_ids: set[str], indexed_ids: set[str], limit=0.10):
    missing = indexed_ids - source_ids          # deleted upstream
    unindexed = source_ids - indexed_ids        # never processed
    if len(missing) > limit * len(indexed_ids):
        raise SystemExit(
            f"refusing to delete {len(missing)} of {len(indexed_ids)} docs; "
            "this looks like a truncated listing, not a deletion")
    return missing, unindexed

The guard is the important line. A source system that returns an empty or truncated listing — an expired token, a paging bug — looks exactly like everything having been deleted, and a reconciler without a limit will faithfully empty your index in one run. Cap the destructive side, require a human to confirm past the cap, and delete through tombstones so the operation is reversible while you find out which it was.