The Data Wall: Are We Running Out of Text?

“We are running out of training data” is a claim with four variables in it, and published estimates differ mostly because they make different choices about two of them. The arithmetic is worth doing yourself.

The claim, stated carefully

The precise version is: the quantity of high-quality human-generated text available for training is finite and growing slowly, while the quantity consumed by frontier training runs has been growing quickly, so at some point the second exceeds the first and the compute-optimal scaling recipe can no longer be followed as written.

Two things it does not claim. It does not say models stop improving — it says one particular input stops scaling, and the programme responds by substituting other inputs. And it does not apply uniformly: text is the constrained resource, whereas video, audio, sensor data and synthetic environments are constrained by processing cost rather than by supply.

The best-known formal treatment is the work by Villalobos and colleagues at Epoch AI on projecting the stock of human-generated public text against training-set growth. It is worth reading in the original for one reason above all: the paper is explicit about which of its inputs are estimates and how wide the resulting range is, and the range is what gets lost when the conclusion is quoted.

How a supply estimate is built

Every such estimate is the same product of four terms. Writing it out makes clear which arguments are about facts and which are about definitions.

usable tokens  =  S  ×  A  ×  Q  ×  E

S  stock of human-generated text that exists at all
A  fraction accessible — crawlable, licensable, not paywalled,
   not legally excluded, not behind a login
Q  fraction surviving quality filtering and deduplication
E  effective epochs — how many times the same token can be
   trained on before it stops contributing

S is estimated from indexed web content, digitised book and journal corpora, code repositories, and user-generated platform content, usually with a growth rate attached derived from internet population and posting rates. It is the term people think the argument is about.

A is a legal and commercial term, not a technical one. It moves with licensing deals, robots.txt policy, paywall adoption and copyright litigation — see training data and copyright for the mechanisms. A large share of the world’s text sits inside private repositories, enterprise systems and messaging platforms, and whether any of that is counted changes the answer enormously.

Q is the largest and least discussed reduction. Deduplication alone removes a very substantial fraction of a raw crawl, and quality filtering removes more. Where the threshold sits is a design decision that trades quantity against quality, and different labs place it differently, so Q is partly a fact about the lab.

E is the one that turns a hard wall into a soft one. The Chinchilla-style recipe implicitly assumes roughly single-epoch training on fresh tokens. Work on data-constrained scaling by Muennighoff and colleagues studied repeating data explicitly and found that several epochs can be close to as useful as fresh data before returns fall away sharply. Multiply the usable stock by that factor and the arrival date of the constraint moves substantially.

Where the uncertainty actually lives

Not in S. Estimates of how much text exists differ, but not by the amount that drives the disagreement. The uncertainty is concentrated in A and Q, and both are decisions rather than measurements — which is why two careful analysts can reach answers an order of magnitude apart without either making an error.

There is a second-order effect that estimates rarely include and which points the other way. As models are deployed, more of the newly created text on the public internet is model-generated, which lowers the growth rate of genuinely human S and simultaneously makes Q harder to apply, because distinguishing human from generated text at scale is unreliable — see AI text detection for why. This is a real mechanism with a well-studied failure mode attached to it, discussed below.

Five ways around it, and what each costs

The pattern across all five: every escape route works best where correctness can be checked automatically, and works worst where it cannot. That asymmetry predicts uneven progress — faster in mathematics, code and formal reasoning than in domains where a good answer is a matter of judgement. It is a prediction about the shape of capability gains rather than their size, and it is one of the few predictions in this cluster that is already partly observable.

There is also a change of regime hidden in the list, and it is easy to miss because the term “data” covers both sides of it. Pretraining consumes a fixed stock: text that already exists, which is finite, non-renewable at the rate required and shared between everyone training a model. Interaction and verified generation consume a flow: data manufactured on demand, limited by the compute available to produce and check it, and private to whoever produced it. A field moving from the first to the second does not simply postpone the constraint — it converts a supply problem into a compute problem, and the constraints that then bind are the ones in compute trends rather than the ones on this page. That is a good reason to be careful with the phrase “running out of data”: the input that becomes scarce is not necessarily the input that matters next.

What would show the wall is real

• Frontier training-set sizes plateauing while compute keeps rising — the signature of a supply constraint rather than a budget one.
• An observable shift in where gains come from: more from post-training, inference-time compute and algorithmic efficiency than from larger pretraining corpora.
• Licensing becoming a competitive differentiator, which is a market signalling that A binds.
• Published evidence of quality degradation traceable to synthetic data in the mix, rather than to any other cause — which is hard to establish and correspondingly rare.

The honest summary is that the constraint is real and its date is uncertain by years, that the escape routes are real and their effectiveness is uncertain, and that anyone quoting a specific exhaustion year without stating their A and Q assumptions is quoting a number, not making an argument.