Compute Trends: Is Exponential Growth Continuing?

Every argument about AI trajectories rests on a compute trend, and four different quantities get called by that name. They have different growth rates, different measurement problems and different implications.

Four quantities, one phrase

A claim that compute growth is slowing, or accelerating, is uninterpretable until you know which of the four is meant. They can move in opposite directions at the same time — frontier run sizes can plateau while total deployed compute grows quickly, which is what you would expect if inference demand rises while training budgets consolidate.

How training compute is estimated

Almost every published training-compute figure is a reconstruction, not a disclosure. There are two standard methods and they should agree.

METHOD 1 — from the model
  C  ≈  6 · N · D
     N = parameters, D = training tokens
     the 6 counts forward and backward passes per parameter
     per token, and is an approximation, not an identity

METHOD 2 — from the hardware
  C  =  chips × peak FLOP/s × seconds × utilisation
     utilisation is the achieved fraction of peak, and is
     the term that is guessed

The first requires knowing the parameter count and the token count, neither of which is reliably published for frontier models any more. The second requires knowing the cluster size, the run duration and the achieved utilisation. The same 6ND arithmetic appears in regulation, where a training-compute number determines whether obligations attach — the mechanics and the ways around it are in compute thresholds.

How wrong those estimates can be

• Utilisation is a guess with a wide range. Achieved throughput as a fraction of peak varies substantially with model architecture, parallelism strategy, interconnect and software maturity. An estimate that assumes a high figure and one that assumes a conservative one differ by a large multiple on the same run.
• Precision changes the units. Peak FLOP/s figures are quoted at a numeric precision, and lower precision yields a larger number for the same silicon. Comparing a modern low-precision figure with an older higher-precision one is comparing different quantities unless it is stated, which it often is not. See quantization for what the precision levels actually are.
• The 6 in 6ND is approximate. It ignores attention cost that scales with sequence length, mixture-of-experts routing where only some parameters are active per token, and any recomputation strategy. For sparse models in particular, using total rather than active parameters changes the answer substantially.
• Boundaries are drawn differently. Does the figure include failed runs, hyperparameter search, ablations, data generation and post-training? Those can together rival the headline run, and different analysts include different subsets.
• Disclosure has decreased. Earlier models came with papers stating architecture and data scale. Recent frontier releases typically do not, so the later points in any long series are estimated with less information than the earlier ones — which biases the shape of the curve in ways that are hard to correct.

Where the published series come from

The systematic work here is worth going to directly rather than through summaries. Sevilla and colleagues assembled a database of training compute for notable machine learning systems over several decades and identified distinct eras with different growth rates; Epoch AI maintains and extends that database publicly, with per-model estimates and stated confidence in each.

Two properties make it usable rather than merely citable. Each entry carries the method and the uncertainty, so you can see which points are documented and which are inferred. And it is maintained, so the recent end of the series is revised as information arrives — which matters, because the recent end is exactly the part any forecast depends on and exactly the part with the least disclosure.

Anyone using such a series should check three things: whether the growth rate is computed over the whole period or over a recent window, since the eras differ; whether the model set is the frontier or all notable systems, which are different populations; and whether trailing points have been revised since the version they are quoting.

There is a selection effect in every such database that no maintainer can remove, and it is worth naming because it biases the curve in a known direction. Entries exist for systems somebody published or announced. Runs that were started and abandoned, runs whose results were disappointing, and internal systems never described in public are absent. The visible series is therefore a record of successful, publicised training runs, which is not the same as a record of compute spent — and the divergence between the two grows as disclosure falls. For the specific question of whether growth is continuing, that matters: a plateau in the published frontier is consistent both with growth genuinely slowing and with the same growth continuing behind a thinner disclosure regime, and the database alone cannot distinguish them. Financial and semiconductor data can, imperfectly, which is one reason the total-deployed-compute quantity is worth tracking alongside the headline one.

What binds next

Extrapolating any of the four curves eventually runs into a physical or economic limit, and the interesting question is which arrives first. These are not predictions with dates attached; they are the constraints that would produce a bend.

• Power. Large training clusters need power delivered to one site, and grid connections have lead times measured in years and are decided by processes unrelated to AI. Discussed in datacentre energy.
• Capital. Each generation costs more than the last, and the return has to justify it. This is a market discipline rather than a physical ceiling, and it can bind earlier than physics does — see frontier training cost.
• Fabrication. Leading-edge capacity and advanced packaging are concentrated in very few facilities, and capacity expansion takes years. The demand-side view is in the accelerator shortage.
• Data. Compute-optimal scaling requires tokens in proportion to parameters, so a data limit is a compute limit in disguise — see the data wall.
• Memory bandwidth. Arithmetic throughput has grown faster than the bandwidth to feed it, so useful compute is increasingly bounded by data movement rather than by FLOP/s. The mechanism is in memory bandwidth.

This page is marked as dated on purpose. The four quantities and their measurement problems are durable; any statement about the current growth rate is not, and should be checked against the maintained databases rather than against a page.