Training Compute vs Inference Compute: Where the Crossover Is

Training a frontier model is the expenditure that gets reported. Serving it is the expenditure that never stops. The point at which the second passes the first is a short calculation, and the answer is smaller than most people assume.

Two formulas

Both are standard approximations used throughout the scaling literature, and both count only the dominant matrix-multiply work, which is what makes them clean enough to reason with.

Training:    C_train  ~=  6 * N * D     FLOPs

  N = parameters, D = training tokens
  The 6 is 2 for the forward pass, plus roughly 4 for the
  backward pass, which computes gradients with respect to
  both the inputs and the weights.

Inference:   C_token  ~=  2 * N        FLOPs per token

  One multiply and one add per parameter, forward only.

The 2 in both is the multiply-accumulate pair. Everything else in the model — normalisation, activations, the attention score computation at modest context — is a smaller correction, and attention becomes significant only when the context is long relative to the model’s width.

The factor of 6 is worth unpacking, since it is the part most often taken on faith. The forward pass performs one multiply-accumulate per parameter per token, which is the 2. The backward pass computes two separate quantities: the gradient with respect to the layer’s inputs, so the signal can continue backwards, and the gradient with respect to the weights, so they can be updated. Each of those costs about as much as the forward pass, giving roughly 4, and 2 + 4 = 6. The same decomposition explains why a forward-only pass is a third of the cost of a training step, and why inference is cheap per token while training is not.

The crossover, derived

Set cumulative inference compute equal to training compute and solve for the number of tokens served, T:

2 * N * T  =  6 * N * D

           T  =  3 * D

The parameter count cancels entirely. The crossover depends
only on how many tokens the model was trained on.

That is a strikingly simple result. A model trained on 15 trillion tokens has consumed as much compute in serving as in training once it has processed about 45 trillion tokens — counting input and output alike, since both pass through the model.

Put a scale on it. If such a model serves a million requests a day averaging 1,000 prompt tokens and 300 generated tokens, that is 1.3 billion tokens a day, and 45 trillion arrives in roughly 95 years. At a billion requests a day it arrives in about five weeks. The crossover is therefore not a property of the model at all — it is a property of how widely the model is used, and the same checkpoint can sit on either side of it depending on deployment.

That range is the actual finding, and it settles an argument that is usually conducted without numbers. A research model, a fine-tune, or any model with a modest user base will never approach its crossover; its lifetime compute is overwhelmingly the training run, and optimising inference for it is close to pointless. A model deployed behind a consumer product at global scale passes the crossover in weeks, after which every further token is compute that dwarfs what it cost to create. Both kinds of model exist, they are often the same architecture, and advice that does not say which one it is about is not advice.

What the clean derivation leaves out

• Training compute is not one run. The reported figure is the final run. Ablations, failed runs, hyperparameter searches and restarts after hardware faults are all real compute, and they are multiples rather than percentages.
• Utilisation differs sharply between the two. Training is compute-bound and large-batch, so it realises a substantial fraction of peak FLOPs. Decode is memory-bound, so achieved FLOPs are a small fraction of peak. Comparing raw FLOPs therefore understates inference’s cost in hardware-hours, sometimes by an order of magnitude.
• Reasoning models multiply the inference side. A model that generates a long chain of intermediate tokens before answering can consume many times the output tokens per user-visible answer, and every one of them costs 2N. This is the largest single change to the inference side of the equation in recent practice.
• Prefill and decode bill differently. Both are 2N per token in FLOPs, but prefill uses the hardware efficiently and decode does not, which is exactly why providers price input and output tokens differently.
• Sparse models break the parameter symmetry. For a mixture-of-experts model, use active parameters in the inference term and total parameters in the memory budget. The crossover derivation still holds with N read consistently.

Why the shift changes what gets built

Once serving dominates the lifetime compute of a model, the optimisation target moves. Training-optimal scaling asks how to spend a fixed training budget best; a serving-dominated world asks how to minimise total cost, and that argues for training smaller models on more data than the training-optimal ratio, because a smaller model is cheaper on every token it will ever serve.

It also explains where engineering effort goes. Quantisation, distillation, speculative decoding, KV cache compression, sparse activation and prompt caching are all attacks on the 2N term or on the bytes moved per token. Each is worth roughly its saving multiplied by every token the model will ever produce — which, past the crossover, is the larger number.

And it reframes what a hardware fleet is for. Training is bursty, tolerant of scheduling, and can run wherever capacity exists, because nobody is waiting on any individual step. Serving is continuous, latency-sensitive and geographically constrained by where the users are. As the second grows relative to the first, the shape of the infrastructure changes: more locations, smaller units, more emphasis on memory capacity and bandwidth than on peak arithmetic, and a much larger penalty for idle hardware. That is a different procurement problem wearing the same word, and mistaking one for the other is how organisations end up with fleets optimised for the workload they used to have.