Synthetic Data for Fine-Tuning: Recipes and Risks

Generating training data with a model is now the default way to build a fine-tuning set, and the case against it is usually cited as a single scary word. The research is more specific than the word, and the specifics tell you how to do it safely.

Four published recipes

What separates the good recipes from the bad is where the constraint comes from. Self-Instruct constrains with a similarity filter, Evol-Instruct with an explicit difficulty operator, rejection sampling with an external verifier. A pipeline that just asks a model for a thousand examples has no constraint at all, and it shows.

What the collapse research showed

The result everyone cites is Shumailov et al., AI models collapse when trained on recursively generated data, published in Nature in 2024. Models are trained on data generated by the previous generation of model, repeatedly. Over successive generations the tails of the distribution disappear first, then the distribution narrows toward its mode, and eventually output degenerates.

The mechanism is not mysterious and does not require anything to go wrong. Sampling from a model produces the common cases far more often than the rare ones. Fitting a model to those samples produces a distribution with less tail than the original. Sample from that and the effect compounds. Each generation is fitted to an approximation of an approximation, and statistical noise plus finite sampling means the errors accumulate in one direction.

Accumulate, do not replace

Here is the part that gets lost, and it is the part that determines your actual practice. The collapse setup replaces the training data at each generation: generation N trains only on data from generation N−1. Follow-up work — notably Gerstgrasser et al. (2024, Is Model Collapse Inevitable?, arXiv 2404.01413) — examined the alternative where synthetic data accumulates alongside the original real data instead of replacing it, and reported that this avoids the collapse observed in the replacement regime.

That distinction maps directly onto what you should do:

• Keep the real data in the mix. Never train generation N+1 exclusively on generation N’s output. Real examples are the anchor and they should stay in every training set.
• Keep real prompts. Even when responses are generated, drawing prompts from production traffic keeps the input distribution anchored to reality — which is the distribution that actually matters at inference time.
• Do not iterate on your own outputs. One generation from a stronger teacher is distillation. Repeatedly training on your own model’s output is the replacement regime, and it is the same trap described in the production-logs page, arriving by a different route.

Diversity is the failure you can measure

Long before anything resembling collapse, synthetic datasets fail in a boring and detectable way: they are repetitive. A model asked for a thousand customer support scenarios will produce a thousand variations on four scenarios, with the same names, the same opening clause and the same rhetorical shape.

Checks worth running on any generated set before training on it, all cheap:

• Distinct n-gram ratio. Unique trigrams divided by total trigrams across the corpus. Compare against a human-written set of the same size. A large gap is your answer.
• Opening-token histogram. Count the first five tokens of every generated response. If a handful of openings cover most of the corpus, you are about to train that opening into the model as a verbal tic.
• Embedding cluster count. Embed everything, cluster, and count how many clusters hold 90% of the mass. A thousand examples in six clusters is six examples with noise.
• Length distribution. Generated data is characteristically uniform in length. Compare the histogram against real traffic; if it is much narrower, the fine-tuned model will inherit that narrowness.

The fix for all four is the same: increase the entropy of the conditioning, not the sampling temperature. Give the generator structured variation to work from — an explicit list of domains, roles, difficulty levels, input formats — rather than hoping a higher temperature invents it.

The risks that are not statistical

• Teacher errors become permanent. A factual mistake the teacher makes consistently is generated consistently, survives any filter that is not checking facts, and is trained in. Rejection sampling against a verifier is the only structural defence.
• Evaluation contamination. If your generator has seen a public benchmark, generated data can contain its items, and your evaluation on that benchmark stops meaning anything. Keep a private held-out set that no generator has touched.
• Licence contamination. Terms of service on the teacher can restrict what you may do with its outputs. Stanford released the Alpaca dataset under CC BY-NC 4.0 for exactly this reason — it was generated from a commercial model whose terms constrained downstream use. That constraint then travels with the dataset to anyone who uses it. See the licensing page.
• Distribution mismatch. Generated prompts are what a model imagines users write. Real users write worse, shorter, with typos, in the middle of a task, with implicit context. A model tuned on imagined prompts is tuned for a population that does not exist.