What a Migration Does to an Existing Synthetic Data Generation Pipeline

You changed the model that generates your synthetic training data, nothing else. The new batch reads well. Then the classifier trained on it scores worse than the one trained on last quarter’s batch, on the same held-out human-labelled evaluation set. The generation prompt never changed, so the prompt is not where you should look first.

The symptom: the data looks fine, the model gets worse

The failure is quiet because synthetic data is reviewed by reading a sample, and a sample of fifty rows from a bad batch reads exactly like a sample of fifty rows from a good one. Every individual row is plausible. What changed is the shape of the whole set: how the labels are balanced, how long the examples are, how many of them are near duplicates of each other, and how much of the input space they cover. A downstream model does not learn from rows, it learns from the distribution, and a distribution can degrade without any row looking wrong.

The mechanism is that a generation prompt is only half of a generator. The other half is the model’s own defaults — how it interprets “varied”, what it reaches for when asked for a negative example, how long it makes something when you did not say. Those defaults are not in your prompt file, they are not in version control, and they are exactly what a migration replaces. Your generator has one undeclared dependency and you just upgraded it.

Four statistics that move when the generator changes

These are the four worth computing on both batches before you conclude anything. All four are cheap and none of them requires the new model.

• Label balance. If your prompt asks for a mix of classes without pinning the proportions, the mix is the model’s choice. Count the classes in both batches. A generator that used to produce roughly even positives and negatives can drift toward whichever class its instruction-following now considers the more helpful answer.
• Length distribution. Compare the median and the 90th percentile character count per field. A model that calibrates response length to perceived task complexity will produce longer examples for the open-ended prompts in your set and shorter ones for the narrow prompts, changing the shape rather than the mean.
• Near-duplicate rate. Embed every row and count pairs above a fixed cosine threshold, or use a cheap MinHash pass if you want to avoid the embedding cost. This is the statistic that moves most and the one nobody checks. See embedding-based deduplication for the mechanics.
• Vocabulary coverage. Type-token ratio over the whole batch, plus the count of tokens that appear in the old batch and not the new one. A collapse here means the generator has settled into a narrower register.

Why diversity collapses first

Near-duplicate rate is the first thing to go, and there is a structural reason. A prompt of the form “generate 100 varied customer complaints about billing” asks the model to be its own diversity source across a single generation. Two things commonly change under a migration and both attack that. First, sampling parameters: several current model families reject temperature, top_p and top_k outright, so a pipeline that got its variety from a temperature setting loses that lever entirely and has to get variety from the prompt instead. Second, a model that follows instructions more literally will interpret “varied” against its own idea of the space rather than sampling widely across it.

The fix is to stop asking the model for diversity and to supply it. Put the axes of variation in the calling code as an explicit cross product — customer tenure band, billing product, emotional register, channel — and generate one example per cell, with the cell values interpolated into the prompt. The model then has one narrow job per call, the coverage is a property of your loop rather than of the model, and it survives the next migration unchanged. This costs more calls and is the single highest-value change you can make to a synthetic pipeline.

The gate that belongs in front of the new batch

Compute the four statistics for the last known-good batch, store them as a JSON baseline next to the dataset, and fail the generation job when the new batch moves outside a band you set. This is the same pattern as a drift baseline on a regression suite, applied to data rather than to outputs.

# gate.py — run after generation, before the batch is promoted
import json, statistics as st

def profile(rows):
    lens = [len(r["text"]) for r in rows]
    labels = {}
    for r in rows:
        labels[r["label"]] = labels.get(r["label"], 0) + 1
    n = len(rows)
    return {
        "n": n,
        "label_share": {k: v / n for k, v in labels.items()},
        "len_median": st.median(lens),
        "len_p90": sorted(lens)[int(0.9 * n)],
        "dup_rate": near_duplicate_rate(rows, threshold=0.93),
        "type_token_ratio": type_token_ratio(rows),
    }

base = json.load(open("baseline.json"))
new = profile(load_batch("batches/2026-08.jsonl"))

FAIL = []
for label, share in base["label_share"].items():
    if abs(new["label_share"].get(label, 0) - share) > 0.05:
        FAIL.append(f"label {label}: {share:.2f} -> {new['label_share'].get(label, 0):.2f}")
if new["dup_rate"] > base["dup_rate"] * 1.5:
    FAIL.append(f"dup rate {base['dup_rate']:.3f} -> {new['dup_rate']:.3f}")
if new["len_median"] > base["len_median"] * 1.4 or new["len_median"] < base["len_median"] * 0.7:
    FAIL.append(f"median length {base['len_median']} -> {new['len_median']}")
if new["type_token_ratio"] < base["type_token_ratio"] * 0.85:
    FAIL.append("vocabulary coverage collapsed")

if FAIL:
    raise SystemExit("generator drift:\n  " + "\n  ".join(FAIL))

The thresholds above are starting points, not findings. Set yours by computing the same profile across two consecutive batches from the old generator: that spread is your natural batch-to-batch noise, and anything wider than it is signal.

What to re-validate downstream

Passing the gate means the batch is comparable, not that it is good. Two further checks earn their cost. Train the downstream model on the new batch and score it against a human-labelled held-out set — not a synthetic one, because a synthetic evaluation set generated by the same model shares the same blind spots and will happily agree with itself. And if any part of your labelling is done by a model rather than a human, note that you have migrated the judge at the same time as the generator; separate those two changes and land them one at a time, or you will not know which one moved the number.

There is one more comparison worth running before you accept the batch, and it costs nothing because you already have the artefacts. Score the old downstream model on the new synthetic data and the new downstream model on the old data. Four numbers instead of two, and they separate two hypotheses that a single comparison confounds: a batch that is genuinely worse will drag down whichever model is trained on it, while a batch that is merely different — a distribution the evaluation set does not represent — shows a more mixed pattern. That distinction decides whether you fix the generator or fix the evaluation set, and getting it backwards costs a week.

Finally, record the generating model in the dataset metadata alongside the prompt version. A synthetic dataset whose provenance says only “generated 2026-04” cannot be debugged after the fact, and this is the cheapest possible time to add the field. When someone asks in six months why the March data behaves differently, the answer should be in the file rather than in a chat log.