How Big the Training Runs Are: Deriving Training FLOPs

10 min read · updated August 4, 2026

Training compute for a language model is estimable to within a small factor from two numbers that are usually public: how many parameters it has and how many tokens it saw. The estimate is six times their product, and it is close enough that the labs themselves use it.

The estimate

A training step does a forward pass and a backward pass. The forward pass costs about two floating-point operations per parameter per token, for the reason set out in the FLOPs-per-answer derivation. The backward pass computes gradients with respect to both the activations and the weights, which is about twice as much work again.

  C  ~=  6 * N * D

  C = total training compute, in FLOPs
  N = number of parameters
  D = number of training tokens

  6 = 2 (forward) + 4 (backward)

This is the estimator used throughout the scaling-laws literature and it is accurate to roughly the tens of per cent for dense transformers trained conventionally. It does not include failed runs, hyperparameter searches, ablations or the compute spent on the data pipeline, all of which are real costs that no paper reports and which can be a large multiple of the headline figure.

Two checks against published figures

An estimator is worth exactly as much as its agreement with cases where the true value is known. Two model papers report their own training compute, so the derivation can be checked against them.

Check 1 — GPT-3.

Brown et al., "Language Models are Few-Shot Learners" (2020) describes a
175-billion-parameter model trained on 300 billion tokens, and reports a
total training compute figure of about 3.14e23 FLOPs.

  6 * 175e9 * 300e9
    = 6 * 5.25e22
    = 3.15e23 FLOPs

Agreement to better than 1%.


Check 2 — Llama 3 405B.

Meta's Llama 3 paper (2024) describes a 405-billion-parameter model
trained on roughly 15.6 trillion tokens, and reports a training compute
figure of about 3.8e25 FLOPs.

  6 * 405e9 * 15.6e12
    = 6 * 6.318e24
    = 3.79e25 FLOPs

Agreement to better than 1%.

Two independent cases, five years and two orders of magnitude apart, both landing within one per cent. That is why this estimate is treated as good enough for regulatory thresholds rather than as a back-of-envelope guess.

The parameter and token counts in those checks come from the papers themselves and should be read there rather than taken from here. Both papers are freely available on arXiv under the titles given.

The other route: GPU-hours

Many model cards report accelerator-hours instead of FLOPs, which is a different and equally usable route. It requires one extra assumption: what fraction of the hardware’s peak throughput the run actually achieved, conventionally called model FLOPs utilisation.

  C  =  gpu_hours * 3600 * peak_flops_per_second * MFU

Worked on Llama 2 70B. Meta's paper reports roughly 1.7 million
A100-80GB hours for that model, trained on about 2 trillion tokens.
NVIDIA's published A100 specification gives a dense BF16 peak of
312 TFLOP/s.

  Route A, from GPU-hours, assuming 40% MFU:
    1.7e6 * 3600 * 312e12 * 0.40
      = 6.12e9 * 312e12 * 0.40
      = 1.91e24 * 0.40
      = 7.6e23 FLOPs

  Route B, from 6ND:
    6 * 70e9 * 2e12  =  8.4e23 FLOPs

The two routes agree to about 10%, which back-solves an implied MFU of
roughly 44% — a plausible figure for a large well-tuned run, and a
useful consistency check on both numbers.

Published MFU figures for very large training runs have generally sat between about 35 and 50 per cent; Google’s PaLM paper (Chowdhery et al., 2022) reported a figure in the mid-forties and introduced the metric in its current form. If your two routes disagree by more than a factor of about two, one of the inputs is wrong, and it is usually the token count.

Why 10^25 is now a legal number

Training compute stopped being a technical curiosity when regulators began writing it into law. The EU AI Act uses cumulative training compute as the trigger for its general-purpose-model obligations: a model is presumed to carry systemic risk when the compute used for its training exceeds 10²⁵ floating-point operations. That makes the estimate above the arithmetic behind a compliance question.

Which models cross 10^25 under 6ND?

  6 * N * D  >  1e25
        N * D  >  1.67e24

  70e9 params x 15e12 tokens  = 1.05e24   ->  6.3e24 FLOPs   under
 405e9 params x 15e12 tokens  = 6.08e24   ->  3.6e25 FLOPs   over
   8e9 params x 15e12 tokens  = 1.20e23   ->  7.2e23 FLOPs   well under

At 15 trillion training tokens, the threshold is crossed at roughly:

  N  =  1.67e24 / 15e12  =  111 billion parameters

Two consequences follow that are worth stating plainly. Training a smaller model on more tokens moves you towards the threshold just as surely as making it larger, because the product is what counts. And “cumulative” is doing real work in the legal text: continued pretraining and large fine-tuning runs add to the total rather than starting a new one. The obligations themselves are covered in the EU AI Act page, and the threshold’s design in compute thresholds.

Where the published estimates live

The model papers and model cards themselves. The only primary source. Look for a table of parameters, tokens, accelerator-hours and emissions; most open-weights releases carry at least two of the four.
Epoch AI’s database of notable AI models. A maintained dataset of training-compute estimates that publishes its estimation method and confidence for each entry, which is the part that matters — an estimate whose derivation you cannot see is not usable as a citation.
The Stanford HAI AI Index. Reproduces compute trend charts annually with attribution to their original sources, which makes it a convenient index rather than a source.

Frontier training compute rises quickly, so any specific model’s position in the distribution ages within months. The derivation on this page does not age; the examples in it do. Check the current frontier against Epoch’s database rather than against a number in an article.

What the estimate cannot tell you

It is not the cost of the project. The published run is the last one. Failed runs, ablations and hyperparameter sweeps are unreported and can exceed the final run several times over.
It says nothing about capability. Compute predicts loss reasonably well under fixed conditions and predicts usefulness badly. Data quality, post-training and preference tuning move perceived quality far more than the last doubling of compute.
It is unreliable for sparse models. For mixture-of-experts training, N should be the active parameter count per token, and the routing and load-balancing machinery adds overhead the estimator does not model.
Token counts are frequently unstated or ambiguous. “15 trillion tokens” may mean unique tokens or tokens seen including repeats over multiple epochs. Those are different numbers and only the second belongs in 6ND.
It tells you nothing about inference. A model trained once is served billions of times; over a popular model’s life, total inference compute can exceed training compute. That comparison is the subject of training versus inference compute.