Re-Embedding a Codebase Incrementally on Every Commit

Re-embedding on commit is a small script, and almost every published version of it is wrong in the same three ways: it ignores deletions, it never fires on a merge, and it has no way to tell you whether it worked. This one handles those.

What the hook has to do

The job is to take the index from describing commit A to describing commit B, doing work proportional to the difference between them. That decomposes into five operations, and a hook that performs fewer than five leaves the index subtly wrong.

• Delete the chunks belonging to files removed between A and B.
• Re-chunk every added or modified file and compute each chunk’s content hash.
• Skip any chunk whose hash is already in the vector store — this is what makes a formatting-only or rename-only commit nearly free.
• Embed and insert the remainder, in batches.
• Record B as the index watermark, and only after everything above succeeded.

The ordering of the last step is the part that matters. If you write the watermark first and the embedding call then fails, the index claims to describe a commit it does not, and every subsequent incremental update starts from a false baseline — the missing chunks are never revisited, because nothing will ever diff across that range again. Write the watermark last and a crash costs you one repeated batch, which is harmless.

Getting the change set from git

Git computes the change set for you and gives it a status letter per path. Rename detection is on by default for diff-tree in most configurations but is worth requesting explicitly, because a detected rename is the difference between two embedding calls and zero.

# what changed in the commit just made
git diff-tree --no-commit-id --name-status -r -M HEAD

# A  src/billing/retry.py        added
# M  src/billing/client.py       modified
# D  src/legacy/old_client.py    deleted
# R096  src/util.py  src/common/util.py    renamed, 96% similar

# what changed since the index's watermark, for catch-up
git diff --name-status -M <indexed_sha> HEAD

The second form is the one that makes the whole design robust. If the hook ever fails to run — the developer used a GUI that skips hooks, the machine was offline, someone force-pushed — the next run diffs from the recorded watermark rather than from the previous commit, and catches up automatically. Build the hook around the watermark diff and use HEAD~1..HEAD only as the degenerate case of it.

The hook, step by step

1. Create .git/hooks/post-commit, make it executable with chmod +x, and have it do nothing but invoke your indexer so the logic lives in a versioned file rather than in an untracked hook.
2. Read the watermark. Store it in a file the repository ignores — .git/code-index/head is a good place, because it is per clone and never committed.
3. Diff from the watermark to HEAD with rename detection, and filter the result to the extensions and paths you index. Exclude vendored and generated directories here, once, rather than in four other places later.
4. For deleted and renamed-from paths, remove the corresponding rows from the file and chunk tables and tombstone their vectors.
5. For added and modified paths, read the blob at HEAD — with git show HEAD:path, not from the working tree, which may already contain newer uncommitted edits — chunk it, and hash each chunk.
6. Query the vector store for which of those hashes already exist. Embed only the misses, batching a few hundred chunks per request.
7. Insert the new vectors, then write HEAD to the watermark file as the last action.

#!/usr/bin/env bash
# .git/hooks/post-commit  — non-blocking wrapper
set -euo pipefail
root="$(git rev-parse --show-toplevel)"
mkdir -p "$root/.git/code-index"
# detach: a commit must never wait on a network call
nohup python "$root/tools/index_sync.py" \
  >> "$root/.git/code-index/sync.log" 2>&1 &
exit 0

# tools/index_sync.py  — the part worth reading
import subprocess, pathlib, hashlib

ROOT = pathlib.Path(subprocess.check_output(
    ["git", "rev-parse", "--show-toplevel"], text=True).strip())
MARK = ROOT / ".git" / "code-index" / "head"
EXTS = {".py", ".ts", ".tsx", ".go", ".rs", ".java"}
SKIP = ("vendor/", "node_modules/", "generated/")

def git(*args):
    return subprocess.check_output(["git", *args], cwd=ROOT, text=True)

head = git("rev-parse", "HEAD").strip()
base = MARK.read_text().strip() if MARK.exists() else None

if base is None:                       # first run: index everything
    entries = [("A", p) for p in git("ls-files").splitlines()]
else:
    entries = []
    for line in git("diff", "--name-status", "-M", base, head).splitlines():
        parts = line.split("\t")
        status = parts[0][0]           # R096 -> R
        if status == "R":
            entries.append(("D", parts[1]))
            entries.append(("A", parts[2]))
        else:
            entries.append((status, parts[1]))

def indexable(p):
    return pathlib.Path(p).suffix in EXTS and not p.startswith(SKIP)

for status, path in entries:
    if not indexable(path):
        continue
    if status == "D":
        store.delete_chunks(path)       # tombstone, then remove rows
        continue
    blob = git("show", f"{head}:{path}")
    for chunk in chunk_by_symbol(blob, path):
        cid = hashlib.sha256(
            f"{MODEL}\x00{CHUNKER_V}\x00{chunk.text}".encode()).hexdigest()
        if not store.has(cid):
            pending.append((cid, path, chunk))

for batch in batched(pending, 256):
    store.insert(batch, embed([c.text for _, _, c in batch]))

MARK.write_text(head)                   # last, always last

The hooks people forget to install

post-commit alone covers a fraction of the ways a working tree’s content changes. It does not fire on git merge when the merge is a fast-forward, it does not fire on git checkout of another branch, it does not fire on git pull, and it does not fire on git rebase for the commits being replayed.

Install the same script behind post-merge, post-checkout and post-rewrite as well. Because the script diffs from a stored watermark rather than from HEAD~1, running it four times for one operation is harmless — the second run finds no difference and exits. That idempotence is why the watermark design is worth the extra file. A hook that assumes exactly one new commit cannot be installed on post-merge at all without double-counting or missing work.

Why this should not block the commit

A hook that calls an embeddings API synchronously adds network latency to git commit, and it will eventually add a timeout to it. Developers respond to that by passing --no-verify or deleting the hook, and then the index silently stops updating for that person — which is a worse outcome than a slow commit, because it is invisible.

There is a second reason to detach, which is that the work is bursty in a way that interacts badly with rate limits. A rebase of forty commits fires the hook forty times in a few seconds; a merge of a long-lived branch produces one change set of two thousand files. If the indexer embeds synchronously it will hit a tokens-per-minute ceiling in both cases, and the natural response — retrying inside the hook — turns a two-second commit into a two-minute one. Push the pending chunks onto a durable queue and drain it at a fixed rate, and both bursts flatten out. The watermark then advances when the queue drains rather than when the hook returns, which is the correct semantics anyway: the index is current when the vectors are stored, not when the diff was computed.

Detach the work, as the wrapper above does, and make failure observable somewhere other than the terminal: the watermark file falling behind HEAD is itself the health signal, and a check that compares the two is the cheapest possible monitor. That comparison is also the fix for an index that has silently gone stale, and the storage side of the same design is covered in the index architecture page.