Testing Backward Compatibility When You Add a Field to an Output Schema

Adding a field to a schema feels like the safest change available. It is safe in one direction and not in the other, and which direction bites you depends on a validator setting you probably chose eighteen months ago and have not thought about since.

Compatibility has two directions

Backward: can a consumer written against the old schema still read a document produced under the new one? It can, if the consumer ignores unknown fields. It cannot, if the consumer validates with additionalProperties: false — a setting that is both good practice and, in the case of structured outputs, mandatory on the request side. That is the collision at the heart of this page.

Forward: can a consumer written against the new schema read a document produced under the old one? Only if the new field is genuinely optional in the consumer’s validator and in the code that reads it. A new required field breaks every stored document you have, which matters the moment anything replays history — a re-scoring job, an analytics query, a cache warmed before the deploy.

Both questions have to be asked, because a change can pass one and fail the other, and because the two failures show up in different places. Backward failures appear at the boundary between services; forward failures appear in batch jobs, days later.

Under strict schemas, nothing is optional

The rule that makes this harder than ordinary API versioning is that strict structured outputs require every field to be listed as required. OpenAI’s guide says so directly — all fields must be specified as required — and suggests emulating an optional field by making its type a union with null. So “add an optional field” is not a thing you can do. What you can do is add a required field whose value may be null, which is a different change with a different consequence: every output now carries the key, including the ones where it means nothing.

That is fine as long as your consumers distinguish absent from null, and many do not. A consumer that treats a present key as a signal — if ("refund_reason" in payload) — starts firing on every document the day the field is added, with a null value. This is a real bug class rather than a hypothetical, and it is invisible to a schema validator because the document is valid. The test has to assert on consumer behaviour, not only on validity.

A corpus of recorded outputs

You cannot test compatibility against outputs you generate fresh, because those come from the new schema and prove nothing about the old. Keep a directory of real recorded outputs, one file per case, committed and never regenerated wholesale. Twenty is plenty; they only need to cover the shapes, not the volume.

Record them with their schema version in the filename, and add to the corpus whenever a production output surprises you. A corpus that grows from incidents is worth more than one generated from the schema, because the interesting documents are the degenerate ones — the empty array, the null in the field somebody swore was always populated, the unicode in the identifier.

The test

// compat.test.ts
import { readFileSync, readdirSync } from "node:fs";
import { join } from "node:path";
import Ajv from "ajv";
import { describe, expect, it } from "vitest";
import { oldSchema, newSchema } from "../src/schemas";
import { readPayload } from "../src/consumer";

const ajv = new Ajv({ allErrors: true, strict: false });
const validateOld = ajv.compile(oldSchema);
const validateNew = ajv.compile(newSchema);

const DIR = join(__dirname, "fixtures/outputs");
const corpus = readdirSync(DIR).map((f) => ({
  name: f,
  doc: JSON.parse(readFileSync(join(DIR, f), "utf8")),
}));

describe("schema change: adding settlement_note", () => {
  it("every stored document still validates against the new schema", () => {
    for (const { name, doc } of corpus) {
      expect.soft(validateNew(doc), name + ": " + ajv.errorsText(validateNew.errors))
        .toBe(true);
    }
  });

  it("a new-shape document is still readable by the old consumer", () => {
    const produced = { ...corpus[0].doc, settlement_note: null };
    expect(validateOld(produced)).toBe(true);
  });

  it("the consumer does not treat a null field as present", () => {
    const withNull = { ...corpus[0].doc, settlement_note: null };
    expect(readPayload(withNull).hasSettlementNote).toBe(false);
  });
});

The second test is the one that fails if the old consumer validates strictly, and that failure is the correct outcome — it is telling you the change is not backward compatible for that consumer and needs the rollout ordering below. Do not fix it by loosening the old validator in the test; that changes the test into a description of a system you do not have.

Ordering the rollout

1. Relax the consumers first. Every reader that validates strictly gets deployed with the new field permitted — and only permitted, not required — before anything starts producing it.
2. Wait for that to be everywhere, including the batch jobs, the replay tooling and any consumer you do not own. This is the step that gets skipped, because the producers are the interesting change.
3. Change the producing schema and start emitting the field, still treating a missing value as legitimate everywhere it is read.
4. Only after the corpus of stored documents has been backfilled or aged out should anything treat the field as reliably present. Until then it is a field that is sometimes null and sometimes absent, and code that conflates those two is the bug this whole sequence exists to avoid.

The same ordering applies to removals, reversed: stop producing, wait, then stop accepting. And it applies to the model as much as to your code — a prompt that instructs the model to populate a field it has never seen before will populate it inconsistently at first, so run the new schema against your golden dataset and check the fill rate before you let anything downstream depend on it.

One structural decision makes all of this cheaper, and it is worth making before the first schema change rather than during the third. Version the schema explicitly and carry the version in the document — a schema_version field the producer sets and every consumer reads. It converts an ambiguous document into an unambiguous one: a reader that encounters a version it does not recognise can fail loudly instead of guessing, and a corpus fixture carries its version with it rather than in a filename convention that erodes. The cost is one integer per output; the benefit is that “absent because the model did not fill it” and “absent because this document predates the field” stop being the same observation.

What this does not buy you is protection from the model changing its mind about how to fill an existing field. Compatibility testing is about the shape of the contract, and a field whose type is unchanged but whose values shift — a category that starts being emitted in title case, a currency that starts arriving with a symbol — passes every test on this page. That is the province of quality regression testing, and the two suites should not be merged: one is deterministic and runs in milliseconds, the other is statistical and does not.