The Chinchilla Paper and Compute-Optimal Training

Before this paper, everyone was building models that were too big and training them on too little data. After it, the ratio changed across the industry. It is the clearest case in recent memory of a single empirical result redirecting an entire field’s spending — and of a conclusion being stretched well past what it supports.

The question it asked

Training Compute-Optimal Large Language Models (2022) asks a narrow, well-posed question. You have a fixed training compute budget. Compute is roughly proportional to parameters multiplied by training tokens, so you can spend it on a larger model trained briefly or a smaller model trained for longer. Which allocation gives the lowest final loss?

The prevailing practice at the time, informed by earlier scaling work, put most of the additional budget into parameters. Models grew far faster than the corpora they were trained on. The paper’s finding is that this is wrong: for a compute-optimal allocation, parameters and training tokens should be scaled roughly in equal proportion. Double the budget and you should roughly increase both by the same factor, not pour it all into size.

Note what kind of question this is. It is not “what is the best model”. It is “given this constraint, what is the optimal split”. Nearly every misuse of the result comes from forgetting the constraint.

Three ways of answering it

The methodological choice worth studying is that the paper does not answer the question once. It answers it three times, by different routes, and reports that the three broadly agree:

• Fix the model size, vary the number of training tokens. Train each size for several durations, read off the loss at each compute level, and find the minimum across sizes.
• Fix the compute budget, vary the model size. The isoFLOP approach: for a given budget, sweep sizes and find the bottom of the resulting curve. This one is the most visually convincing, because the curve has a clear minimum you can point at.
• Fit a parametric loss function. Fit a functional form for loss as a function of parameters and tokens to all the runs, then minimise it analytically under a compute constraint.

Convergent evidence from methods with different failure modes is the strongest structure an empirical paper can have, and it is worth recognising when you see it. If the three approaches had disagreed, the interesting paper would have been about why. That they agreed is what licensed the confident headline — and it is also why the later challenge, discussed below, was to one method rather than to the conclusion.

The demonstration

Rather than stopping at curves, the paper trains a model to test the prediction. Chinchilla is a 70-billion-parameter model trained on substantially more data than was then usual — around 1.4 trillion tokens — using the same compute budget as Gopher, a 280-billion-parameter model trained on far fewer tokens. Same budget, four times smaller, four times more data. Chinchilla outperformed Gopher across the great majority of the evaluation suite.

This is the design to internalise, because it is a genuine controlled comparison and they are rare. One quantity is held fixed and made explicit, one is varied, and the prediction is registered in advance by the theory the paper just built. Contrast that with the usual shape of a result: a new model, better numbers, and no way to attribute the improvement to any particular decision. That is what the ablation section exists to fix and what a controlled comparison does natively.

There is a second consequence the paper draws out and that is easy to skip: a compute-optimal smaller model is also cheaper to serve, forever. Training compute is spent once; inference compute is spent on every request. So the result was not merely academically tidy — it changed the economics of deployment, which is why it was adopted so fast.

Three misreadings

“Twenty tokens per parameter is the right ratio”

The roughly twenty-to-one figure that circulates is the optimum for training loss under a fixed training budget. It is not a law about the best model to deploy. If you plan to serve a model billions of times, the total-cost optimum sits at a smaller model trained on far more tokens than compute-optimal — you deliberately overspend on training to underspend on inference forever. Several later model families are trained far past the compute-optimal point for exactly this reason, and they are not violating the paper; they are optimising a different objective.

“It means bigger models do not help”

It means nothing of the kind. Holding compute fixed, a smaller well-trained model beats a larger under-trained one. Increase the compute and the optimal model gets bigger too. The result is about allocation, not about a ceiling.

“Data quality does not enter into it”

The analysis counts tokens. It does not model what is in them, and the paper is not a claim that all tokens are interchangeable. Curation, deduplication and mixture ratios sit entirely outside the framework, and the practical reason the ratio has drifted since is partly that the tokens available are not the tokens the fit was made on.

What was later contested

A replication attempt published in 2024 examined the third of the three approaches — the parametric fit — and reported that the fitted coefficients given in the paper could not be recovered from the paper’s own reported data, and that the confidence intervals on those coefficients were implausibly narrow. The reanalysis proposed revised estimates that sat closer to the paper’s other two methods than to its own third one.

The right way to hold this is the interesting part. It is not a debunking: the central conclusion — that parameters and tokens should scale together, and that the models of the era were badly under-trained — came from three independent routes, and the challenge touched one. But it is a real correction, it came from outsiders working from published figures, and it is a demonstration of the thing this cluster keeps insisting on: a headline can be right while a specific number under it is not, and the only way anyone finds out is if the method and the data are reported well enough to be checked.

That is also why the same scrutiny cannot be applied to the frontier systems that followed. When a technical report withholds the training compute, the token count and the parameter count, there is no fit to re-examine and no reanalysis to publish. More on that pattern in what reproduction actually requires.