Scaling Hypothesis vs Missing Ingredients

This is the central technical disagreement in the field, and it is usually conducted between two positions that neither camp holds. Nobody serious believes scale alone with no other changes; nobody serious believes scale has stopped mattering.

What the hypothesis actually claims

The strong scaling hypothesis, as held by its actual proponents, is roughly: the capabilities that remain missing are not blocked by an absent conceptual ingredient, and will be obtained by continuing the current programme — more parameters, more and better data, more training compute, more inference compute, and the ordinary engineering progress that accompanies all of it. Architecture and training-recipe improvements are expected and are part of the programme; the claim is that no new paradigm is required.

That last qualification is where the argument really is, and it is slippery in a way both sides exploit. If a future system uses reinforcement learning on verifiable tasks, a retrieval component and a memory system, is that scaling the current paradigm or replacing it? Proponents count it as scaling because the core learned function and the training machinery are continuous with today’s. Critics count it as evidence that the missing ingredients were real and had to be added. Both readings are available for the same facts, which is why the debate resists resolution by any single result.

The weak version — that scale reliably improves performance on a broad range of tasks — is not in dispute by anyone and is not what the argument is about.

The case for, at full strength

• The empirical regularity is unusually clean. Loss follows a predictable power law in parameters, data and compute over many orders of magnitude. Kaplan and colleagues established the form in 2020; Hoffmann and colleagues corrected the parameter-to-data ratio in 2022 with the Chinchilla results, which changed how frontier runs are sized. Regularities that hold across that range in a messy empirical field are rare and are reasonable things to extrapolate from.
• Capabilities that were argued to need special mechanisms appeared without them. Multi-step arithmetic, code synthesis, translation between low-resource pairs, in-context learning of a new task from examples: each was at some point offered as an example of something the approach could not do, and each arrived through more of the same. Rich Sutton’s Bitter Lesson generalises the pattern across the history of the field — general methods that leverage computation have repeatedly beaten methods that build in human knowledge.
• Nothing has been shown to be impossible. The arguments for missing ingredients are arguments from current failure, and current failure has a poor record as a predictor of permanent failure in this field specifically.
• The programme keeps finding new axes. When pretraining data became the binding constraint, inference-time compute became a productive new dimension — see test-time compute for how much can be bought there. That is what a healthy paradigm looks like from the inside.

The case for missing ingredients, at full strength

• Loss is not capability. Scaling laws predict loss on the training distribution. The mapping from loss to the abilities anyone cares about is not itself a law, and the fact that a regularity is clean does not tell you what it is a regularity in.
• Sample efficiency is off by orders of magnitude and the gap is not closing. Systems require vastly more data than humans for comparable competence. François Chollet’s argument in On the Measure of Intelligence is that this is not a detail but the definition of the thing that is missing, and the ARC benchmarks were built specifically to measure it. The details are in sample efficiency.
• No deployed system learns continuously. Weights are frozen at deployment, so a system does not incorporate what it encountered yesterday. Whether this is an engineering choice or a capability gap is genuinely open — see continual learning — but every human comparison assumes an ability current systems lack.
• Data is finite and the good data is scarcer. A programme that requires ever more human text runs into supply, and the substitutes have their own problems. See the data wall.
• Yann LeCun’s architectural argument. His position, set out in the 2022 Path Towards Autonomous Machine Intelligence proposal, is that predicting tokens in text cannot produce the kind of predictive model of the physical world that planning requires, and that a different objective — predicting in a learned representation space rather than in the observation space — is needed. This is a specific technical claim with a specific proposed alternative, not a general scepticism.
• Progress measurement is unreliable in a known direction. Contamination and benchmark saturation both inflate apparent progress, so some of the evidence for scaling is measurement artefact. How much is disputed; that some is, is not.

What both sides agree on

Considerably more than the public argument suggests, and the agreements are the useful part.

• Scale has produced large, real capability gains. The dispute is about what remains, not about what happened.
• Data quality and curation matter as much as quantity at current scales.
• Benchmark results overstate real-world capability, for reasons that are documented rather than speculative.
• Current systems are unreliable on long-horizon tasks in ways that are not simply a function of single-step accuracy.
• Compute, energy and capital constraints will bind eventually. The disagreement is about when relative to the capabilities of interest.

A dispute where the parties agree on this much of the evidence is usually a dispute about extrapolation rather than about facts, and extrapolation disputes are resolved by waiting or by finding a discriminating measurement — not by argument.

What would move each side

The question worth putting to anyone holding either position: what would you expect to see if you were wrong?

Both columns describe things that could be observed within a few years rather than at some indefinite horizon, which makes this one of the more tractable disagreements in the cluster. It is also worth holding the possibility that both are partly right: that scale carries the programme a long way and that one or two of the named ingredients turn out to be genuinely necessary. That outcome would look, in retrospect, like both sides having been correct about the half they emphasised — which is the usual way paradigm disputes actually end.