Fine-Tuning Run Planner

Estimates the hours, device-hours and memory a fine-tuning run needs from dataset size, epochs, model size and the hardware figures you supply.

Training examples

Tokens per examplePrompt plus completion, after templating.

Epochs

Model parametersTotal, in billions. For a mixture-of-experts model use the active count and treat the answer as a floor.

FLOPs per parameter per token6 for full fine-tuning: 2 forward, 2 for input gradients, 2 for weight gradients. About 4 with LoRA, where the frozen weights need no weight gradient.

Device throughputPeak dense TFLOP/s at your training precision. Example value — take it from your own accelerator's datasheet.

TFLOP/s

Model FLOPs utilisationShare of peak actually achieved. Well-tuned large runs are commonly reported in the 0.3–0.5 range; a first attempt is lower.

Devices

Rate per device-hourExample figure — use the rate you are actually quoted.

$ / h

Bytes per parameter, weights and optimiser stateMixed-precision full fine-tuning with Adam is about 16 (2 weights + 2 gradients + 12 states and master copy). LoRA over a frozen fp16 base is about 2.

bytes

VRAM per deviceExample value — put your own accelerator's memory in.

GiB

Wall clock for the run

2.5 h

10.0 device-hours, $20.00 at the rate above. State and weights come to 29.80 GiB per device — inside the 80 GiB you entered, before activations.

Training tokens — examples × tokens × epochs: 90,000,000
Total FLOPs — factor × params × tokens: 4.32e+18 FLOP
Per device, after utilisation: 1.20e+14 FLOP/s
Across the cluster: 4.80e+14 FLOP/s
Wall clock: 2.5 h
Device-hours: 10.0
Cost at the rate above: $20.00
Training throughput: 10,000 tokens/s
Weights and optimiser state, total: 119.21 GiB
Per device, sharded evenly: 29.80 GiB
Fits the VRAM entered: yes, before activations

What this assumes: the 6ND form of the compute estimate — six FLOPs per parameter per token for a dense forward and backward pass — with attention's quadratic term ignored, which is safe at short sequence lengths and increasingly optimistic as they grow. Scaling across devices is assumed linear; it is not, and communication overhead makes a multi-node run slower than this. Memory counts weights, gradients and optimiser state only: activations, the KV cache during evaluation, and fragmentation are extra, and they are what actually causes the out-of-memory error. Data preparation, checkpointing, failed runs and the hyperparameter search that precedes the real run are not in here, and on a first fine-tune they cost more than the run does.

Compute is predictable; memory is what stops you

Training compute is one of the few genuinely forecastable quantities in machine learning. A dense forward and backward pass costs about six FLOPs per parameter per token — two forward, two computing gradients with respect to the inputs, two computing gradients with respect to the weights — so the total is that factor times parameters times tokens seen. Divide by what your hardware actually delivers, which is peak throughput times a utilisation factor well below one, and you have the hours.

The factor is where LoRA shows up, and not in the way people expect. Freezing the base weights removes the weight-gradient term, roughly six FLOPs to four: a saving, but not a large one, because you still have to backpropagate through the whole frozen network to reach the adapters. LoRA's real win is the memory column. Full mixed-precision fine-tuning with Adam carries something like sixteen bytes per parameter once gradients, moments and a master copy are counted, and for an eight-billion-parameter model that is over a hundred gigabytes before a single activation. The same model with a frozen fp16 base and a small adapter is a fraction of that. That is why a fine-tune that could run in hours does not run at all, and why the memory row usually decides the plan rather than the time row.

Treat the utilisation and throughput fields as the honest sources of error. Both are properties of your specific setup, both move by a factor of two between a naive and a tuned configuration, and the answer moves with them.

Fine-Tuning Run Planner

Compute is predictable; memory is what stops you

The reasoning behind this

Related tools