Sample Efficiency: Humans vs Models
5 min read · updated August 3, 2026
Everyone agrees models need far more data than children. The size of the gap depends entirely on decisions about what to count, and those decisions move the answer by more than the disagreement they are meant to settle.
Why there is no single number
The comparison sounds like it should reduce to a ratio: tokens seen by a model, words heard by a child, divide. Four things prevent that.
- The human figure is an estimate from small observational studies of speech in the home, extrapolated across years. Those studies vary substantially between families and between methodologies, and the best-known of them have been the subject of replication disputes.
- A child’s input is not only words. It is words attached to objects, faces, actions and consequences, arriving in a causally structured stream the child partly controls. Counting only the words counts the smallest part of it.
- The comparison target is undefined. A five-year-old and a language model are not competent at the same things in the same way, so “comparable competence” needs a task before it means anything.
- Whether evolutionary optimisation counts as training is a real question with no neutral answer, and it changes the result by orders of magnitude on its own.
The consequence is that a stated ratio is mostly a report of the author’s counting choices. That is why this page gives the choices instead.
Five ways to count
| Counting choice | Description |
|---|---|
| words only | Words a child hears before some age against tokens in a training corpus. The most common comparison and the most favourable to the human side, because it counts the model's full input and only a fraction of the child's. |
| all sensory input | Count the visual, auditory and proprioceptive stream too, at some bit rate. The human number becomes enormous and the gap narrows or reverses. The obvious objection is that most of that stream is redundant — and quantifying the redundancy is itself unresolved. |
| task-relative | Fix a task and count what each needs to reach the same performance. Well defined, and the answer varies hugely by task: models are extremely efficient at some pattern-completion tasks and extremely inefficient at learning a new physical skill. |
| in-context | Compare few-shot learning within a session against a human learning a new rule from a few examples. On this measure the gap is small or absent, which is why in-context learning surprised people. The counter is that this ability was itself purchased with the pretraining corpus. |
| lifetime including evolution | Treat the genome as the product of an optimisation process and count its cost as pretraining. The human number becomes astronomical, and the model looks efficient. Whether this is legitimate is the subject of the next section. |
These are not five estimates of one quantity. They are five different quantities, and citing one as the sample-efficiency gap without saying which is where most of the confusion originates.
The priors problem
A newborn is not a randomly initialised network. Considerable structure is present at birth — the organisation of sensory cortex, biases toward faces and speech, mechanisms for tracking objects — and it was not learned by that child. Where it came from is a long optimisation process over an enormous population and timescale.
Two readings, and both are held by serious people.
Evolution as pretraining. Learning is learning regardless of substrate, and the genome encodes the result of a search whose cost belongs in the comparison. On this reading, comparing a model’s pretraining to a child’s lifetime is comparing a full process to a fine-tuning run.
Priors as architecture. What evolution supplies is mostly structure — a learning algorithm and an architecture — rather than content, and the information content of the genome is small compared with any training corpus. On this reading, evolution is analogous to the choice of architecture and optimiser, which nobody counts as training data.
Chollet’s framing in On the Measure of Intelligence takes the disagreement seriously and offers a way through it: define intelligence as skill acquisition efficiency relative to stated priors and stated experience, so the priors are declared rather than argued about. The ARC benchmarks operationalise this by using tasks that rely only on knowledge assumed to be innate — objectness, counting, basic geometry — so the prior is fixed by construction and the measurement is of what is learned from the few examples given.
The comparison done properly
The cleanest way to answer an unfair comparison is to make it fair, and that is what the BabyLM Challenge does. It restricts training to a corpus on the scale of what a child might plausibly hear in the first years of life, then evaluates the resulting models on standard linguistic benchmarks. Everything else — architecture, curriculum, objective — is open.
Two things make it the most useful evidence in this area. It converts a rhetorical comparison into a shared, reproducible benchmark with a fixed budget. And it turns the question from “how big is the gap” into “what closes it”, which is answerable: entries can test whether curriculum ordering, multimodal grounding, different objectives or architectural priors help under a fixed data budget.
The related theoretical work — Michael Frank’s analysis of the data gap between children and language models is a good entry point — is careful to describe the comparison as approximate and to enumerate exactly the confounds listed above. That care is the model to follow when reading any figure from this literature.
One result from that line of work deserves stating on its own, because it is easy to lose in the framing. Models trained on child-scale corpora acquire a great deal of grammatical competence — enough that the interesting failures are elsewhere. What they do not acquire at that scale is the breadth of world knowledge and the robustness on reasoning tasks that large-corpus models have. That is informative about where the data is actually going: much of the enormous corpus is buying facts and coverage rather than the structure of the language, and a sample-efficiency argument aimed at grammar lands differently from one aimed at knowledge. It also suggests the gap is not one gap. Whatever makes a child efficient at acquiring syntax may be a different mechanism from whatever would let a system acquire a new domain of knowledge from a handful of examples, and lumping both into a single ratio hides that.
Why the gap matters for the bigger arguments
Sample efficiency is not a curiosity; it is load-bearing in three arguments elsewhere in this cluster.
- It is the sceptics’ strongest empirical card. If a system needs many orders of magnitude more data for comparable competence, something about how it learns differs qualitatively, and that is the argument in scaling versus missing ingredients.
- It interacts with the supply constraint. Poor sample efficiency is what makes finite text a problem at all. Improve efficiency enough and the ceiling in the data wall moves out of the way.
- It is entangled with continual learning. Part of what makes human learning look efficient is that it accumulates — each new thing is learned against everything learned before, and never re-derived. A system that cannot accumulate pays full price every time. See continual learning.
The defensible summary: on any counting method that restricts the model to a child-scale corpus, the gap is large and real. On counting methods that include the full sensory stream or evolutionary priors, it narrows or reverses. Anyone who reports the gap as a single figure has picked one of those methods, and the first thing to ask is which.