The Data Pipeline Behind Every AI Feature

Every retrieval feature, every extraction job and every fine-tuning run sits on the same pipeline. It is worth drawing once, because the expensive mistakes are all made at the joints between its stages rather than inside any of them.

The eight stages

Between “a document exists somewhere” and “a model can answer questions about it with a citation” there are eight distinct jobs. They are worth naming separately even in a small system, because each fails differently and each has a different cost per document.

Stages 1 and 2 are cheap and idempotent. Stage 3 is where most of the engineering goes and where each format fails in its own way. Stage 7 is where the bill is, because embedding a corpus is priced per token and re-running it is priced again.

The reason to keep them separate even when a single script would do is that they have different reasons to change. Stage 3 changes when you meet a new format or fix a parser. Stage 5 changes when somebody looks at the retrieved chunks and notices the site navigation in them. Stage 6 changes every time anyone reads an article about chunk sizes. Stage 7 changes when a better or cheaper embedding model appears. Four independent change frequencies inside one function means every change is a change to everything, and it is the reason the “just a script” version of this always ends up being rewritten rather than extended.

The contract between stages

Here is the decision that determines whether this pipeline is maintainable, and it is made in the first week: do the stages hand each other values in memory, or artefacts in storage?

The in-memory version is one function that opens a file and returns embeddings. It is shorter, it is easier to test end to end, and it has one property that becomes intolerable at any real size: to redo stage 6 you must redo stages 1 through 5. When you change the chunk size — and you will change the chunk size — you re-fetch and re-parse the entire corpus to get there.

The artefact version writes the output of every stage to durable storage under a key derived from the content hash of its input plus the version of the code that produced it. Stage 6 reads stage 5’s artefacts. Change the chunker, bump its version, and stages 1 to 5 are cache hits.

# The key is a pure function of (what went in, what code ran).
# Same inputs + same code version => same key => already done.

def artifact_key(stage: str, version: str, input_hash: str) -> str:
    return f"{stage}/v{version}/{input_hash[:2]}/{input_hash}.json"

def run_stage(stage, version, fn, input_hash, load_input, store):
    key = artifact_key(stage, version, input_hash)
    hit = store.get(key)
    if hit is not None:
        return hit                      # nothing recomputed, nothing billed
    out = fn(load_input(input_hash))
    store.put(key, out)
    return out

That is the whole idea, and everything else in this cluster assumes it. Content hashing is what makes reprocessing selective, and selective reprocessing is the difference between a chunking experiment costing an afternoon and costing the embedding bill twice.

What that looks like on disk

Three prefixes, one per durable thing, and a database that holds pointers rather than payloads:

raw/sha256/9f/9f3c...bin          original bytes, never modified
text/v3/9f/9f3c....json           extracted + normalised text, offsets
chunks/v7/9f/9f3c....json         segments with [start,end) into text/v3

documents(id, source_uri, raw_sha256, fetched_at, mime, status)
chunks(id, document_id, text_version, chunk_version, start, end, text)
embeddings(chunk_id, model, dim, vector)

Keeping the raw bytes forever is not sentimentality. It is the only way to answer “did the parser change or did the document change?”, and it is what lets you re-extract with a better parser in two years without going back to a source system that may no longer exist.

Note what the database does and does not hold. It holds identifiers, hashes, offsets and status — small, indexed, queryable things — and it points at the large immutable payloads rather than containing them. That split is what keeps the operational database small enough to restore quickly and to run migrations against, while the artefacts sit in object storage where they are cheap and where nothing ever needs to rewrite them. It also means the two can be backed up on different schedules, which is appropriate because one of them never changes.

One further consequence is worth naming because it is the thing people most often wish for later: with artefacts on disk under content addresses, two versions of the pipeline can run against the same corpus at the same time. The new chunker writes under a new version prefix, the old one keeps serving, and the comparison is a query against two sets of rows rather than a story about what happened last month.

Where it actually breaks

• Silent empty extractions. A scanned PDF returns the empty string rather than an error. Without an assertion that extracted text is non-trivial, those documents enter the index as zero-length rows and the feature is quietly missing a slice of the corpus. This is the single most common data bug in retrieval systems and it is three lines to prevent.
• Encoding decided too late. If stage 4 guesses, every downstream artefact inherits the guess. Decide the encoding once, at the moment you first have the bytes and the HTTP headers, and store the decision.
• Chunk identity that is not stable. If a chunk id is an array index, re-chunking renumbers everything and every stored citation now points somewhere else. Derive chunk ids from content.
• No verification stage. Everything up to stage 7 can succeed while the index ends up with documents nobody can retrieve — a filter on a metadata field that was never populated will do it.

Sizing it before you build it

Two numbers decide the shape of the whole thing, and both can be estimated on paper. The first is total tokens: documents × tokens_per_document. A 40-page PDF of prose is on the order of 20,000 tokens, so 50,000 such documents is around 10⁹ tokens. The second is the per-token price of stage 7, which you multiply by the first.

The useful part of that arithmetic is not the total. It is the ratio between the one-off cost of building the index and the recurring cost of querying it, because they push in opposite directions: a smaller chunk size means more embeddings to build and more precise retrieval at query time. Work both out before you pick a chunk size, not after.