When Synthetic Data Fails: Five Failure Modes and Their Signatures

Generated datasets rarely fail loudly. They fail by producing a model that scores well on everything you measured and behaves worse on everything you did not — which is why the check that matters is the one you run before training, not the one you run after.

Why these are always caught late

Three properties of generated data conspire to delay detection. It looks right — every row is fluent, well formed and plausible, so reading a sample gives false confidence in a way that reading a sample of scraped data never does. It is self-consistent, so a model trained on it and evaluated against data from the same generator scores extremely well. And its errors are systematic rather than random, so they do not average out with volume and they do not show up as variance in any aggregate.

Put together: a pipeline can produce a confidently wrong dataset, train a model that measures well against a generated eval, and reveal nothing until a user meets a case the generator never imagined. Every check below exists to shorten that loop.

The five modes

1. Distribution mismatch

The generated inputs are not the inputs your users send. Generated support tickets are grammatical, single-topic, politely phrased and about 60 words; real ones are three words or four hundred, contain two unrelated questions, a pasted stack trace and a screenshot reference. The model learns the task as posed by the generator.

Signature: excellent scores on a generated eval, a large drop on any sample of real traffic, and a length or vocabulary histogram that is visibly narrower than the real one. Check: plot the length distribution of generated and real inputs on the same axes before training. It is the cheapest diagnostic in this cluster and it catches this mode almost every time.

2. Mode collapse in the generator

The corpus has a thousand rows and six underlying ideas. This is the most common failure and the most measurable one — it is the same phenomenon the diversity metrics exist for, and the tail loss that the collapse literature describes as its first stage.

Signature: low distinct-n against a real baseline, a handful of embedding clusters holding most of the mass, and an opening-token histogram dominated by a few prefixes. Check: run the opening-prefix count on any generated corpus before doing anything else with it. Two seconds, no dependencies.

3. The teacher’s errors, learned faithfully

Whatever the generating model gets wrong, it gets wrong consistently, and a student trained on its output learns those mistakes as facts — including the confident, fluent, incorrect ones. This is the mechanism under distillation’s ceiling: a student cannot exceed its teacher on knowledge the teacher lacks, and it can inherit the teacher’s specific misconceptions rather than a random spread of errors.

Signature: the fine-tuned model is wrong in the same way across many inputs, and its errors match the generator’s when you check. Check: hard verification wherever one exists — run the code, check the arithmetic, confirm the extracted value appears in the source. Where verification is impossible, a second model family as reviewer at least decorrelates the errors. Feng and colleagues’ Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification is the published version of this argument.

4. Recursive contamination

Generated data leaks back into the inputs of the next generation. It happens by accident far more often than by design: generated examples are written into the same table as real ones with nothing marking them, production logs now contain model-written text that was pasted by users, and next quarter’s “real traffic” seed corpus is partly synthetic. That is the replacement regime the collapse literature studies, assembled unintentionally.

Signature: diversity metrics that decline slowly across dataset versions, and the fraction of rows nobody can attribute rising over time. Check: a mandatory provenance flag on every row, and a report of the synthetic fraction per dataset version. If you cannot compute that fraction, you cannot detect this mode at all — which is the practical reason provenance is a technical control and not paperwork.

5. Eval contamination by the generator

The same generator, or the same seed documents, produced both the training data and the evaluation data. The eval then measures how well the model learned the generator, which is a question nobody asked. It is the ordinary contamination problem with an extra twist: no string overlap is required, because the shared structure is stylistic and distributional rather than lexical, so an n-gram decontamination pass finds nothing and reports success.

Signature: scores far above what any real-traffic sample produces, and a suspiciously small gap between train and test performance. Check: the evaluation set must be real, held out and created independently of the generation pipeline. That is the entire fix, and there is no substitute for it — build it first, before generating anything.

The pre-training audit

Thirty minutes over any generated dataset, in the order that finds the most for the least effort. Every step answers one of the five modes above.

• Read twenty rows at random. Not the first twenty — random. You are looking for a shared shape, a recurring name, an opening clause. Most bad corpora are diagnosable by eye in four minutes, and people skip this because it feels unrigorous.
• Count the opening prefixes. First five tokens of every row, most common ten. Mode 2.
• Overlay the length histograms. Generated inputs against real inputs, same axes. Mode 1.
• Compare vocabulary. Terms frequent in real traffic and absent from the generated set are the slice the generator does not know exists — often product names, error codes, or slang. Mode 1 again, from the other side.
• Verify a sample against ground truth. Fifty rows, checked by whatever verifier or expert is available. The error rate you find is the error rate of the corpus, and it is a number worth writing into the datasheet. Mode 3.
• Compute the synthetic fraction. Per source, per version, from the provenance flags. Mode 4.
• Confirm the eval’s independence. Different provenance, different seeds, ideally different people. Mode 5.

The one control that catches all five

If only one thing survives from this page: keep a fixed, real, held-out set that no generated data ever touches, and evaluate every model against it, every time.

It is a small set — the arithmetic of a small eval set says what it can and cannot resolve, and the answer is that it resolves category-level breakage very well, which is exactly what these failure modes produce. Real inputs mean a distribution mismatch shows up as a score gap. Independent creation means the generator cannot flatter itself. Never regenerating it means the comparison across months is meaningful, which is what makes slow recursive contamination visible at all.

Every failure mode on this page is invisible when the only measurement is against data from the same pipeline that produced the training set. One honest instrument, kept fixed, is worth more than every other check here combined — and unlike the rest of them, it keeps working after everyone has stopped paying attention.