How Much Text There Is to Train On, and How It Is Counted

Dataset sizes are quoted in bytes, in documents and in tokens, and only one of those is a property of the text alone. The other two depend on choices, which is why two honest people can report different sizes for the same corpus.

Three units, routinely confused

That last row is the source of most published disagreement. The same text tokenised with a 50,000-entry vocabulary yields substantially more tokens than with a 200,000-entry one, so a corpus can be “200 billion tokens” and “280 billion tokens” at the same time, both correctly. See how tokenizers differ for why the ratio moves.

Bytes to tokens, derived

You can convert between the units well enough for any sizing purpose, and you can measure the constant yourself in about a minute rather than borrowing it.

  tokens  ~=  bytes / bytes_per_token

For English prose in UTF-8 with a modern BPE tokenizer of roughly
100k-200k vocabulary entries, bytes_per_token sits near 4.
Older 50k-vocabulary tokenizers land nearer 3.7-3.9 on the same text,
and non-Latin scripts are far lower — which is the whole of the
tokenizer language tax.

Worked:
  1 GB (1e9 bytes) of English prose
    at 4.0 bytes/token  ->  250 million tokens
    at 3.8 bytes/token  ->  263 million tokens

  1 TB of English prose
    at 4.0 bytes/token  ->  250 billion tokens

So a 15-trillion-token English training set implies roughly
  15e12 * 4  =  6.0e13 bytes  =  60 TB of cleaned plain text.

That is cleaned text after deduplication and filtering. The raw crawl
it came from is one to two orders of magnitude larger.

Measure your own constant on your own text. Take a representative megabyte, tokenise it, and divide. The number you get is the only one that describes your corpus, and it is the same procedure as in the token counts of a book, a codebase and a year of email.

Corpora that state their own size

These are public datasets whose creators published a size in the paper or the dataset card. They are worth knowing because they are the only large corpora whose contents can be inspected — the training sets of the major closed models are not disclosed at all, which is a fact in itself.

• The Pile (EleutherAI, 2020). Described in its paper as an 825 GiB dataset assembled from 22 named sub-corpora, each of which is listed with its own size. The first large corpus whose composition was fully published, and still the best worked example of what a mixed training set is made of.
• C4 (Google, 2019), the Colossal Clean Crawled Corpus, derived from one Common Crawl snapshot with a documented filtering procedure. Its value is the filter description, which became the template most later pipelines modified.
• RefinedWeb (TII, 2023), released alongside the Falcon models, arguing that heavily filtered web text alone can match curated mixtures. Publishes both its size and its filtering steps.
• Dolma (Allen Institute for AI, 2024). Released with an open toolkit for the filtering and deduplication pipeline, which makes it the most reproducible of the set.
• FineWeb (Hugging Face, 2024), a large filtered Common Crawl derivative published with ablation experiments showing what each filtering decision did to downstream quality. The ablations are the interesting part.
• Common Crawl itself, which is not a training set but the substrate under most of them. It publishes per-crawl statistics including page counts and language distribution, and those pages are the place to get a current figure rather than a quoted one.

How anyone estimates the total stock

The question “how much text exists” has no census answer, and the serious attempts are all built the same way: take something countable, estimate what fraction of the whole it represents, and divide. Epoch AI’s work on data limits, published under the title Will we run out of data? by Villalobos and colleagues and updated since its 2022 first version, is the best-documented example. It puts the effective stock of public human-generated text on the order of 10¹⁴ tokens.

The method is worth understanding because it shows where the uncertainty is:

1. Start with something enumerable, typically the indexed web via crawl statistics or search-engine index-size estimates.
2. Estimate what share of the real web that enumeration covers. This is the step with the largest uncertainty and it is not measurable directly, only bounded.
3. Convert to tokens with a bytes-per-token assumption, exactly as above.
4. Subtract duplicates. Web-scale deduplication removes a very large fraction, and how large depends on whether you deduplicate exactly or fuzzily — see corpus deduplication.
5. Subtract what quality filters reject, and what is legally or practically unavailable.

Every one of those steps carries a multiplicative uncertainty, so the honest output is a range spanning at least an order of magnitude. Any source that quotes a single figure for the total stock of human text without a range has lost information somewhere between the estimate and the sentence.

Raw is not usable

The gap between crawled bytes and trainable tokens is where most of the data goes, and it is large.

• Boilerplate and markup. Navigation, cookie banners and templates dominate raw HTML by volume. Extraction throws away the majority of the bytes on a typical page.
• Duplication. The web repeats itself constantly: syndicated articles, mirrored documentation, quoted text. Exact and near-duplicate removal is one of the largest reductions in any pipeline and one of the largest quality improvements.
• Language. A multilingual crawl filtered to one language keeps only that language’s share, and the distribution is extremely uneven — the subject of how many languages models really support.
• Quality filtering. Classifier-based filters reject a substantial further fraction, and FineWeb’s published ablations are the clearest public evidence that aggressive filtering improves results rather than merely shrinking the set.
• Contamination removal. Benchmark test sets must be stripped or every evaluation afterwards is invalid. See dataset decontamination.

What running out would actually mean

The arithmetic of the constraint is straightforward. Compute-optimal scaling relates parameters and tokens, so a training-token budget implies a model size, and a fixed stock of text implies a ceiling on the budget. That is the shape of the argument in the data wall, and the Chinchilla paper (Hoffmann et al., 2022) is where the relationship between the two was established.

Three routes around it are already in use, and each has a published literature rather than only a hope: synthetic data, which brings its own failure mode in model collapse; multiple epochs over the same data, which works for a small number of repeats and degrades after; and non-text modalities, where video in particular is vastly larger than all text ever written and vastly less information-dense per byte.

What no reliable public figure exists for is how much text the frontier labs actually hold, including licensed and non-public corpora. It is not disclosed, it is commercially sensitive, and estimates of it are extrapolations from the public web that by construction cannot see the private part.