Schema Versioning When Your Extraction Changes

The schema you shipped in March is not the schema you will want in September. Unlike a database migration, some changes to an extraction schema cannot be applied to existing rows by any transform, and the difference is worth knowing before you have four million of them.

Extracted data is derived data

A row in your extractions table is the output of a function of four inputs: the document, the schema, the prompt and the model. Change any one and later rows are not comparable with earlier ones. Store all four identifiers alongside every record:

extractions
  id
  document_id        -> the source, immutably addressed (content hash, not a path)
  schema_version     integer, monotonic
  prompt_version     integer or a git sha
  model_id           the exact string sent, including any date suffix
  extracted_at
  data               jsonb   -- conforms to schema_version
  raw_response       jsonb   -- what the model actually returned. see below.

Without schema_version you cannot write a migration, because you cannot tell which rows need it. Without model_id you cannot explain the day your accuracy changed and nothing in your repository did — a provider retiring a snapshot behind an alias is a routine event and it is invisible unless you wrote the resolved id down.

Three kinds of change

A fourth case sits outside the table and catches people out: nothing in your repository changed and the output distribution moved anyway, because the provider updated the model behind an alias. That is a version change in the producer with no version change in the code, and it is only detectable if model_id records what the request actually resolved to rather than the alias you asked for. When it happens, the correct response is the same as for a semantic change: treat records either side of the boundary as different populations until you have compared them.

Semantic changes are the dangerous ones precisely because nothing breaks. Every old row still validates against the new schema and every old row now means something different from what it says. The only safe handling is to treat a semantic change as a new field with a new name, or to re-extract everything and refuse to mix the two versions in one query. Halfway measures here produce reports that are quietly wrong for the months that straddle the change.

The migration chain

Structural changes get the same treatment as database migrations: small, numbered, forward-only functions applied in sequence, so a record at any version can be brought to current.

CURRENT = 4

def _1_to_2(d: dict) -> dict:
    d = dict(d)
    d["customer_name"] = d.pop("cust")          # rename
    return d

def _2_to_3(d: dict) -> dict:
    d = dict(d)
    d["currency"] = d.get("currency") or "OTHER"   # new enum, safe default
    return d

def _3_to_4(d: dict) -> dict:
    d = dict(d)
    total = d.pop("total")
    d["total_including_tax"] = float(total)     # string -> number, renamed
    return d

MIGRATIONS = {1: _1_to_2, 2: _2_to_3, 3: _3_to_4}

def upgrade(record: dict) -> dict:
    v, data = record["schema_version"], record["data"]
    while v < CURRENT:
        data = MIGRATIONS[v](data)              # KeyError = a gap in the chain
        v += 1
    return {**record, "schema_version": v, "data": data}

Three rules make this hold up. Migrate lazily on read and persist the result, so a bad migration on a rare record does not take down a batch job at 3am. Never edit a migration once it has run in production — add another. And test each step against a frozen fixture of a real record at that version, checked into the repository; those fixtures are the only surviving evidence of what version 2 looked like.

Store the raw output

The single highest-leverage decision in this whole area, and it costs a few kilobytes per record: keep the model’s complete response, unparsed and untransformed, next to the record you derived from it.

It converts a large class of additive changes from “re-run the model over four million documents” into “run a script”. If you extracted a total and later want currency, and the raw response happened to contain a total_text of "EUR 1.250,00", the backfill is a regex. It also lets you re-derive after a bug in your own transform code without paying for inference twice, which is a more common cause of backfills than schema evolution is.

Keep the reasoning fields, the quotes and the fields you discarded. Space is cheap; a second pass over your corpus at current token prices is not.

The related discipline is to version the prompt as deliberately as the schema, because a prompt edit changes the output distribution just as a schema edit does, and it is far easier to make casually. A one-word change to a field description is a new prompt_version. Keep the prompts in the repository rather than in a dashboard, so the version is a commit and a bisect is possible; a prompt edited in a web UI at 4pm on a Friday is an untracked deployment, and it is the single most common explanation for “the extraction got worse and nothing changed”.

When you must re-extract

Sometimes there is no way around it: the new field is not in the raw response, or the change is semantic. Then it is a cost question, and the cost is documents × input_tokens × price, which for a corpus of any size is worth estimating before you commit rather than after.

• Re-extract in a shadow table. New schema_version, both versions live at once, and a comparison you can look at before cutting over. Extraction is not deterministic; a diff between old and new is data, and it frequently finds a bug in the new prompt.
• Only the affected fields. A narrow schema over the same document is far cheaper on output tokens, and input dominates anyway, so combine it with caching if your provider offers it.
• Prioritise by use. Documents nobody has queried in a year can migrate lazily on first access. Backfilling a whole corpus eagerly is usually a decision made because it feels tidier, not because anything needs it.
• Never mix versions in an aggregate. Any query that spans a semantic change must filter on schema_version or it is producing a number that means nothing. Enforce it in a view rather than in a convention.