Model Collapse: What the Research Actually Showed

Model collapse is cited as though it settles the question of training on generated data. It does not, and the reason is not that the result is wrong — it is that the result is about one specific experimental setup, and most real pipelines are not in that setup.

The result everyone cites

Shumailov and colleagues published AI models collapse when trained on recursively generated data in Nature in 2024, developing an earlier preprint titled The Curse of Recursion (arXiv 2305.17493). The construction is a chain of generations: train a model, sample from it, train the next model on those samples, sample again, and so on. They ran this on language models fine-tuned on a wikitext corpus and, separately, worked it out analytically for simple cases including a Gaussian and a mixture, where the effect can be shown rather than only observed.

What degrades, and in what order, is the useful part. They describe an early collapse in which the tails of the distribution disappear — the rare events stop being represented — and a late collapse in which the distribution has narrowed so far that it bears little resemblance to the original and variance shrinks toward a point. Perplexity on the real data rises generation after generation, and the text becomes repetitive in a way that is obvious to read.

Why it happens, in one paragraph

No malfunction is required. Sampling a finite number of examples from a distribution under-represents its rare regions, because rare things are rare. Fitting a model to that sample yields a distribution with slightly less tail than the one you sampled from. Sample from that, and the loss compounds — each generation is an approximation of an approximation, and the approximation error does not cancel out, because it has a direction. Add functional approximation error and optimisation error on top and the drift is faster. The mechanism is ordinary statistics, which is why it can be derived and not merely observed.

The condition: replacement

Here is the sentence that most citations of this work leave out. In the collapse construction, each generation trains on data that replaces the previous data. Generation N is trained on samples from generation N−1 and not on the original corpus. That is what makes it a recursive process at all.

It is a legitimate model of one scenario: the open web filling with generated text, later models scraping it, and the real human-written fraction shrinking over time. As a model of a pipeline you control, it is a poor fit, because nobody deletes their real training data when they add generated rows. Alemohammad and colleagues reached the same structural conclusion from the image side in Self-Consuming Generative Models Go MAD (arXiv 2307.01850), where fully synthetic loops degrade in quality and diversity, and loops with a sufficient supply of fresh real data at each generation do not.

The follow-up that changed the picture

Gerstgrasser and colleagues asked the obvious next question in Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data (arXiv 2404.01413). They compared the replacement regime with an accumulation regime, where each generation’s synthetic output is added to the existing data rather than substituted for it, and they reported that accumulation avoids the collapse seen under replacement: test error plateaus across generations instead of diverging. They showed it analytically for linear regression and reported the same qualitative behaviour in language-model and other experiments.

Bertrand and colleagues, in On the Stability of Iterative Retraining of Generative Models on their own Data (arXiv 2310.00429), arrive at a compatible condition from the theory side: iterative retraining is stable provided the initial model is good enough and the proportion of real data in each round is large enough. Both of those are conditions you can satisfy on purpose.

So the honest one-line summary of this half of the literature is: recursively replacing real data with generated data degrades a model; accumulating generated data alongside real data, in the settings studied, does not. That distinction is doing all the work, and it is the reason a fine-tuning run that mixes generated examples into a real corpus is not the experiment those papers ran.

The work pointing the other way

It would be convenient to stop there, and it would be a misreading. Dohmatob, Feng, Kempe and colleagues have published a line of work arguing that the damage is not confined to the pure-replacement case. A Tale of Tails: Model Collapse as a Change of Scaling Laws (arXiv 2402.07043) frames the effect as a degradation of the scaling law itself — the curve of loss against data flattens, so more data stops buying what it used to — and Strong Model Collapse (arXiv 2410.04840) argues that even a small fraction of synthetic data within a much larger real corpus can be enough to prevent the expected gains from scaling, rather than merely diluting them.

Seddik and colleagues (How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse) work in the same direction, deriving bounds on how much synthetic data can be tolerated as a function of the setting. And Feng and colleagues, in Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification (arXiv 2406.07515), give the constructive version: verifying or pruning synthetic data — selecting it rather than accepting it — changes the outcome, which puts the emphasis back on the filter stage rather than on the mixing ratio.

These results are not in direct contradiction with the accumulation papers. They are answering different questions — “does the loop diverge?” versus “does the scaling law survive?” — in different regimes, with different definitions of harm. What they jointly rule out is any confident statement of the form “X per cent synthetic is safe”. No such number has been established, and anyone quoting one is extrapolating from a setup that may not be yours.

What this means for a pipeline you own

• Never close the loop. Do not train a model on its own previous generation’s output, iterated. One pass from a stronger teacher to a weaker student is distillation and is a different thing entirely. Iterating on your own output is precisely the replacement regime.
• Accumulate, do not substitute. Real data stays in every training set. This is the single intervention with direct published support behind it.
• Verify rather than mix by ratio. The literature supports selection much better than it supports any particular percentage. A verified synthetic example — one that passed a test, a solver or a schema — is a different object from an accepted one.
• Keep real prompts even when responses are generated. The input distribution is the one your model meets at inference time. Anchoring it to production traffic costs nothing and removes an entire class of drift.
• Watch diversity, not perplexity. Tail loss is the first symptom and it shows up as reduced variety long before it shows up as bad output. The diversity metrics are cheap and they are your early warning.
• Hold out real data, permanently. An evaluation set of real examples that never enters training is the only instrument that can tell you any of this is going wrong. Build one first.