Schema Evolution in AI Pipelines

Upstream will add a field, rename a field, change a type from string to object, and start sending null where it never did. None of that is avoidable. What is avoidable is finding out about it from a crash in a consumer at three in the morning.

Three schemas, not one

A document pipeline has three separate schemas and they evolve independently. Conflating them is the root of most of the pain.

• The source schema — whatever the upstream system emits. You do not control it and you frequently are not told when it changes.
• The internal schema — your normalised document and chunk representation. You control this completely; it should change rarely and deliberately.
• The index schema — the fields the search store knows about and can filter on. Changing this often means a rebuild, which is why it is worth being conservative about what goes in it.

The mapping between source and internal is the shock absorber. Put the tolerance there — one adapter per source, each responsible for producing a valid internal document or failing loudly — and the rest of the pipeline never sees an upstream change at all.

The internal schema deserves one more constraint that is easy to state and unpopular to enforce: it should contain nothing that is specific to one source. The moment a field exists because a particular vendor sends it, every other adapter has to decide what to put there, and the “internal” schema has become a union of external ones. Put source-specific data in a namespaced extras map instead, where it is available to anyone who wants it and mandatory for nobody.

The compatibility modes, and what they permit

Streaming systems formalised this years ago and the vocabulary transfers directly. Avro’s schema resolution rules and the compatibility modes used by schema registries give you four named answers to “can I deploy this?”:

The transitive distinction is the one that catches people running AI pipelines specifically, because reprocessing the archive is a normal operation here rather than an exotic one. If you will ever re-embed three-year-old documents from their stored artefacts, you need FULL_TRANSITIVE behaviour, not FULL.

The rule underneath all four is the same and it is worth internalising rather than memorising the table: a field added without a default is a breaking change, because a reader encountering old data has nothing to put there. A default turns an incompatible change into a compatible one, which is why every one of the safe operations above has “with a default” attached.

Reading tolerantly

Codify the tolerance at the boundary. Unknown fields are captured, not dropped and not fatal; missing optional fields take defaults; a missing required field fails that document and only that document.

from pydantic import BaseModel, ConfigDict, Field, ValidationError

class SourceDoc(BaseModel):
    # Keep unknown fields instead of dropping them: when upstream adds
    # something useful you have the data already, and when they add
    # something odd you can see it in the extras.
    model_config = ConfigDict(extra="allow")

    id: str
    body: str
    title: str | None = None
    published_at: str | None = None
    tags: list[str] = Field(default_factory=list)

def adapt(raw: dict) -> InternalDoc | None:
    try:
        s = SourceDoc.model_validate(raw)
    except ValidationError as e:
        quarantine(raw, reason=e.json())     # one document, not the batch
        return None
    extras = set(s.model_extra or ()) - KNOWN_EXTRAS
    if extras:
        metrics.increment("source.unknown_field", tags=sorted(extras))
    return InternalDoc(id=s.id, text=s.body, title=s.title or "",
                       published=parse_date(s.published_at), tags=s.tags)

The unknown_field counter is the cheapest early warning in this whole cluster. Upstream adds a field weeks before anyone tells you, the counter moves, and you find out from a metric instead of from a consumer that started failing when the field became mandatory.

Note the granularity of the failure: one document is quarantined and the batch continues. That choice is worth making deliberately, because the alternative — fail the batch on any invalid record — sounds rigorous and behaves badly. A single malformed document from a source of two hundred thousand stops the whole pipeline, and the pressure to get data flowing again produces a bypass flag that then stays on forever. Per-document quarantine with a visible count gives you the same information without the outage.

There is one exception where failing the batch is correct: when the validation failure rate crosses a threshold. One bad document is a bad document; twenty per cent of them failing the same way is an upstream change, and continuing means publishing a corpus that is missing a fifth of itself. That is a corpus-level check rather than a per-record one, which is exactly why both layers exist.

Adding a field to a live corpus

The common request is to add something to every chunk — a language tag, a security label, a section path — over an index that is serving traffic. Doing it in one migration means either a long write lock or a rebuild. The four-step version has neither:

1. ADD the column/field as nullable, no default backfill.
   Postgres 11+ makes ADD COLUMN with a constant default O(1) — it stores
   the default in the catalogue rather than rewriting every row. Adding a
   nullable column has always been cheap. A volatile default is not.

2. WRITE it from the pipeline for every new and updated document.
   Now new data has it and old data does not. Readers must tolerate both,
   which is the FORWARD-compatible position from the table above.

3. BACKFILL in batches, keyed by content hash so it is resumable:
     UPDATE chunks SET lang = $1 WHERE id = ANY($2)   -- 5k ids at a time
   Track progress in a table, not in the shell history.

4. TIGHTEN the constraint only once step 3 reports zero remaining rows.
   This is the step people skip, and skipping it means the field is
   permanently optional and every consumer keeps its null branch forever.

Do not filter on the new field between steps 1 and 4. A query with WHERE lang = 'en' during the backfill silently returns the subset that happens to be done, which is a correctness bug that looks like a relevance problem.

When the change is genuinely breaking

Some changes cannot be made compatible: a type change from scalar to object, a unit change (dollars to cents), a semantic change to a field that keeps its name. For these the only safe move is a new field alongside the old one, both populated during a transition window, and the old one removed when nothing reads it.

Semantic changes deserve special fear because they are invisible to every automated compatibility check. If score was 0–1 and is now 0–100, every schema validator passes and every threshold in your system is wrong. Version the name when the meaning changes — score_pct next to score — and let the compiler and the null checks find the call sites for you. It is uglier and it is the only version of this that has ever worked.

Plan the transition window explicitly rather than letting it run indefinitely. Write down when dual-writing starts, what has to be true before the old field can be dropped — usually a read counter at zero for a full cycle, including whatever monthly job nobody remembers — and who is responsible for the removal. Without that, the transition window never closes: both fields are populated forever, every new consumer picks whichever one it finds first, and the ambiguity you were trying to remove is now permanent and doubled.