What a Frontier Training Run Costs: The Methodology, Not the Headline

Every widely circulated training-cost figure is somebody’s estimate built from three assumptions, and the assumptions do more work than the arithmetic. Here is the arithmetic, with the assumptions marked, so you can build your own estimate and know exactly how wrong it might be.

Why there is no number here

Labs do not publish training costs. What circulates is reconstructed from disclosed or guessed parameter counts, guessed token counts, an assumed hardware efficiency and an assumed hourly rate — and every one of those four is uncertain by a meaningful factor, so the product is uncertain by a large one. Printing a single figure here would launder that uncertainty into a fact and would be stale besides.

What is durable is the method. It is three steps, and the first two are genuinely arithmetic.

The method is worth having even if you will never train anything at this scale, because it is the same method that tells you what a fine-tune will cost, what a continued-pretraining run on domain data will cost, or whether a proposed experiment fits a budget. Substitute smaller values of N and D and every step below works identically. It is also the tool for evaluating a claim: when a figure circulates, the useful response is to ask which parameter count, which token count, which utilisation and which rate were assumed, and a figure that cannot answer those four questions is not an estimate.

Step 1: FLOPs

C = 6 * N * D    floating-point operations

N = parameters
D = training tokens
6 = 2 forward + ~4 backward, per parameter per token

Example: N = 500e9, D = 15e12
C = 6 * 5e11 * 1.5e13 = 4.5e25 FLOPs

The approximation counts the dominant matrix work and omits attention’s quadratic term, which is a modest correction at typical training sequence lengths and grows if a model is trained with very long contexts. For a mixture-of-experts model, use the active parameter count per token here, not the total — that is the entire compute argument for sparsity.

Step 2: accelerator-hours

Peak FLOP/s is never achieved. The standard measure of how much of it a training run realises is model FLOPs utilisation: the FLOPs the model definition requires, divided by peak FLOP/s times elapsed time. Published training reports have quoted MFU in roughly the 30–50% band for large runs — the PaLM paper (Chowdhery et al., 2022) reported 46.2% for its largest configuration, which is a useful anchor precisely because it was published with its method.

accelerator_hours = C / (peak_FLOPS * MFU * 3600)

Example, continuing:
C = 4.5e25
peak = 1e15 FLOP/s at the training precision  [assumption]
MFU  = 0.40                                    [assumption]

hours = 4.5e25 / (1e15 * 0.40 * 3600)
      = 4.5e25 / 1.44e18
      = 31.3e6  accelerator-hours

Wall-clock on 10,000 devices: 3,125 hours ~= 130 days.

Note how sensitive that is. Halving the MFU assumption doubles the answer. Using a peak figure at a lower precision than the run actually used shifts it by another factor of two. Anyone quoting a cost without stating both numbers has not given you an estimate, only a conclusion.

It is worth being clear about why MFU is not close to one, because the gap is not waste in any simple sense. Communication between devices occupies time in which arithmetic is not happening; pipeline stages have bubbles; the optimiser step, data loading and checkpointing all take wall-clock time; and any operation that is not a large matrix multiply — normalisation, activations, the attention softmax — runs well below peak by nature. Larger models and larger batches tend to push MFU up because the matrix multiplies grow relative to everything else, which is one of the quieter reasons scale is efficient.

The wall-clock line in that calculation carries its own constraint. Dividing accelerator-hours by a fleet size gives elapsed time, and elapsed time cannot be shortened indefinitely by adding devices: the global batch size grows with the fleet, and past some point a larger batch stops improving learning per token. That ceiling is why frontier runs take months rather than days regardless of how much hardware is available, and it is a fact about optimisation rather than about procurement.

Step 3: money, which is your input

Multiply accelerator-hours by a rate. This page does not supply one, and the reason is not caution — it is that no single rate is correct. The relevant rate depends on whether the operator owns the hardware, holds a multi-year commitment, or rents on demand, and those differ from each other by large factors. An organisation training a frontier model is almost never paying a published on-demand rate.

compute_cost = accelerator_hours * effective_rate_per_hour

For an owner, effective_rate_per_hour is not a price at all:

  (capex / useful_life_hours) + power + cooling + facility
  + network + staff, all divided by achieved utilisation

Which means the honest form of the answer is a range with the assumptions attached, not a number. If you must present one figure, present the accelerator-hours: it is the part you derived, and it lets your reader apply their own rate.

The multipliers that dwarf the compute

• The final run is not the project. Scaling-law experiments, ablations, data-mixture searches and restarts after hardware failures all consume compute. Reported figures typically describe the successful run only, and the project multiple over it is substantial.
• Failures are structural at this scale. Across tens of thousands of devices running for months, component failures are a certainty rather than a risk. Checkpointing and restart cost real hours, and they show up as reduced effective MFU.
• Data is a cost centre. Acquisition, licensing, cleaning, deduplication and filtering are large expenditures that never appear in a FLOP calculation.
• Post-training is not free. Supervised fine-tuning, preference optimisation and the human or model-generated data behind them are a separate budget, and for models tuned heavily for reasoning it is not a rounding error.
• People. A team capable of running this is a large recurring cost that dominates smaller projects entirely and is never included in a headline figure.

The one durable conclusion: the pure compute term is the easiest to estimate and often the minority of the true cost, which is why reconstructed headline figures should be read as lower bounds on a narrow definition rather than as what the model cost to make.