Memorisation and Generalisation, Measured
12 min read · updated August 4, 2026
“Do language models memorise their training data?” has no answer as asked. What researchers measure is extractability: whether a specific string can be recovered from a model by an attacker with a specific budget, using a specific procedure. That is a different claim, and the difference is the subject of this page.
The question is badly posed
Two intuitions collide here, and both are partly right. A model with billions of parameters trained on trillions of tokens cannot be storing its training set — the arithmetic forbids it. And yet prompt one carefully and it will emit a passage it saw during training, verbatim.
Both are true because “memorised” is doing unstated work. A model can reproduce a string without storing it in any retrieval-shaped way: the string is a high-probability continuation because the pattern was reinforced, and reproduction is generation that happens to land on it. Whether that counts as memorisation depends on what you are trying to decide — a copyright question, a privacy question and a scientific question have different answers.
A rough capacity argument, so the intuition has a number in it.
a 7B parameter model at 16 bits per parameter = 14 GB of weights
its training corpus, at ~2 trillion tokens
and roughly 4 bytes per token of raw text ≈ 8 TB
ratio: the weights are under 0.2% of the corpus size.
Lossless storage of the corpus is impossible by a wide margin. This says
nothing about whether SOME strings are recoverable — a lossy code can
reproduce a small, favoured subset exactly. Which subset, and why,
is the actual research question.Four definitions, four different answers
| Definition | Description |
|---|---|
| verbatim extractable | There exists a prompt of some bounded length such that greedy decoding reproduces the target string exactly. This is what most extraction papers measure. It is relative to the prompt budget and the decoding strategy. |
| discoverable | The string is extractable when you already have a prefix of it. Weaker, and useful for measuring — you can test it systematically over a corpus — but it presumes the attacker already has part of what they are looking for. |
| counterfactual | The model assigns the string much higher probability than an otherwise identical model trained without it. This isolates the contribution of the specific training example, and it is the cleanest definition scientifically — and enormously expensive, because it requires training a second model. |
| membership inference | An adversary can determine whether an example was in the training set, without recovering it. Formalised for ML by Shokri and colleagues in 2017. This is the definition that matters for most privacy law, and it is a lower bar than extraction. |
A paper that reports a high rate under one definition and a headline written against another is the most common failure in coverage of this area. Check which one before you compare two numbers.
What the extraction studies did
Three lines of work, described by their method rather than their numbers.
Carlini and colleagues, 2021. Presented at USENIX Security, this demonstrated training data extraction from GPT-2. Method: generate a very large volume of text from the model, then rank the samples by a membership signal — roughly, sequences the model finds far more likely than a reference model or a compression baseline does — and manually verify the top candidates against the training corpus. Verified extractions included personally identifying information. The important structural point is that extraction was a search problem: the attack was generate-and-filter, and its yield is a function of how much you generate.
Carlini and colleagues, 2022. A follow-up that studied how extractability scales. The reported qualitative relationships are the durable part: extractability increases with model size, increases sharply with how often a sequence is duplicated in the training data, and increases with the length of prefix the attacker is given. Duplication is the strongest of the three, which is why deduplication became the standard mitigation.
Nasr and colleagues, 2023. Extended extraction to deployed, aligned chat models, showing that an attack which pushes the model off its aligned response distribution can cause it to emit training data. The lesson is about alignment rather than about memorisation: instruction tuning suppresses the behaviour without removing the underlying capability, so a prompt that escapes the aligned mode recovers it.
What every extraction number depends on
This is why quoting a rate across contexts is meaningless. Any published figure is a function of at least these:
- The attack budget. Number of generations, prompt length, number of attempts per prompt, and how much verification effort. Double the compute, get a higher rate. A rate without a budget is not a measurement.
- The decoding configuration. Greedy, beam, temperature, and any repetition penalty all change what is reachable. A model can be extractable at temperature 0 and not at temperature 1.
- The candidate set. Rates computed over strings already known to be duplicated in the corpus are not comparable to rates over a random sample of training documents. This choice alone can move a number by orders of magnitude.
- The match criterion. Exact 50-token match, exact 10-token match, and approximate match are three different experiments.
- Deduplication of the training corpus. Model families differ enormously here, and it is one of the strongest determinants of the outcome.
What the results license
Stated carefully, because this is where the reasoning usually breaks.
Supported: some training sequences can be recovered verbatim from some models by an attacker with a realistic budget; heavily duplicated sequences are far more vulnerable than singletons; larger models are more vulnerable at equal duplication; alignment training suppresses the behaviour without eliminating the capability.
Not supported: that the model “contains” its training data in any retrieval sense; that a failed extraction attempt shows a string is absent, since failure is relative to one attack; that an extraction rate measured on one model family transfers to another; or that a low measured rate means a specific document is safe. Absence of extraction is not evidence of absence of the information — it is evidence about the attack you ran, and this is the same asymmetry that makes unlearning so hard to evaluate.
What actually reduces it
- Deduplicate the training corpus. The single most effective known intervention, and the one with the best cost-to-benefit, because duplication is the strongest predictor of extractability. See corpus deduplication for how it is done in practice.
- Differentially private training. The only approach with a formal guarantee, and it costs utility and training efficiency enough that it is rare at frontier scale. The guarantee is per-example, which is the right shape for privacy and the wrong shape for a document duplicated across a thousand sources.
- Output-side filtering. Check generations against a corpus of known-sensitive strings and block matches. Cheap, deployable, and defeated by trivial paraphrase — useful as a backstop, not as a control.
- Do not train on it. Unglamorous and by far the most reliable. Every other option on this list is mitigation of a decision already made.
Reading the next number you see
When a study reports that some percentage of training data can be extracted, five questions decide what it means. Which definition of memorisation? What attack budget produced it? What candidate set was the percentage taken over? What match criterion counted as success? And was the corpus deduplicated? A report that does not answer all five is not comparable to any other report, and the number should not be carried into a sentence about a different model.
This matters beyond research hygiene. These figures end up in litigation, in policy submissions and in arguments about training data and copyright, where a rate detached from its attack budget can support almost any position somebody wants it to.