The Real Cost of a Fine-Tune, End to End

Almost every fine-tuning cost estimate is a GPU-hour calculation, and the GPU hours are usually the smallest of the four terms. This page is deliberately parametric: prices in this field move faster than a page can be revised, so what is offered here is the structure and the derivations, with every figure marked as an assumption you replace.

The four terms

ONE-OFF, per attempt (and you will make more than one attempt)
  1. data        curation hours, labelling, or teacher generation
  2. compute     GPU-hours x rate, x number of runs
  3. evaluation  suite construction + judge tokens + human review

RECURRING, per month, forever
  4. serving     hosting or per-token premium
  4b. maintenance re-training when the base is deprecated,
                 keeping the eval suite alive

total year one ≈ (1 + 2 + 3) x attempts  +  12 x (4 + 4b)

The multiplier on attempts is the term people forget. A first fine-tuning run essentially never produces the model you ship; three to five iterations on data and configuration is normal. Budget the one-off costs as a per-attempt figure and multiply.

There is a second structural reason these estimates come out low, and it is organisational rather than arithmetic. Terms 1 and 3 are mostly people — domain experts writing examples, engineers building evaluation, reviewers adjudicating disagreements — and people-time rarely appears in a proposal that is framed as an infrastructure cost. Term 2 is the only one that arrives as an invoice, so it is the only one that gets counted. Writing the four terms out in the order above, with the recurring ones separated, is most of the work of making the estimate honest.

Term one: data

Usually the largest one-off cost, and the one that appears in no calculator. It takes one of two shapes.

Human curation. Somebody who knows the domain writes or corrects examples. A useful planning rate is minutes per example rather than a headline throughput: simple classification labels are fast, multi-paragraph corrected outputs with a review pass are not, and a realistic figure for the latter is several minutes each including quality control. Five thousand examples at five minutes is over four hundred hours of expert time, and that is a real number that dwarfs any GPU bill.

Teacher generation. Cheaper and computable directly:

generation cost
  = N x (prompt_tokens x input_price + output_tokens x output_price)
    x samples_per_prompt

worked, structure only — substitute current prices
  N = 50,000 prompts
  prompt 500 tokens, output 400 tokens
  4 samples per prompt for rejection sampling
  → 50,000 x 4 x 500 =   100M input tokens
    50,000 x 4 x 400 =    80M output tokens

the multiplier that matters is samples_per_prompt: rejection
sampling at k=4 quadruples this term, and it is usually worth it.

A historical anchor for the ratio between data and compute, useful because it is published and dated: the Stanford Alpaca release in March 2023 reported roughly $500 of API spend to generate 52,000 examples and under $100 to train the 7B model on them. Both absolute figures are long obsolete. The 5:1 ratio between data generation and training is the part that has held up.

Whichever shape you choose, add a review pass and budget it separately. Generated data needs a human to look at a sample and confirm the filters are catching what you think they are catching; curated data needs a second reader to catch the inconsistencies between annotators that would otherwise become contradictions in the gradient. A sample of a few hundred examples reviewed properly is enough, and it is the cheapest quality intervention available at this stage — much cheaper than discovering the problem after two training runs and an evaluation cycle.

Term two: compute, derived

Rather than quote GPU-hour prices that will be wrong, derive the hours from your dataset. The transformer training approximation is C ≈ 6·N·D FLOPs for full fine-tuning; LoRA’s backward pass is cheaper but the forward pass dominates, so it is a safe upper bound.

N = 7e9 parameters
D = 20,000 examples x 800 tokens x 3 epochs = 4.8e7 tokens

C = 6 x 7e9 x 4.8e7                = 2.0e18 FLOPs

accelerator at 3.12e14 FLOP/s peak bf16,
30% MFU on a small single-GPU job  = 9.4e13 FLOP/s

time = 2.0e18 / 9.4e13             = 21,300 seconds
                                   ≈ 6 GPU-hours

x 4 attempts                       ≈ 24 GPU-hours

Twenty-four GPU-hours. Whatever the hourly rate is when you read this, this term is not what decides the project. That is the point of doing the derivation: it stops the conversation from being about GPUs.

Two corrections in the other direction. Quantised training is slower per step — QLoRA buys feasibility on small hardware, not throughput, so multiply the hours if you are memory-constrained. And a 70B model at the same dataset size is ten times the FLOPs and needs multiple accelerators, which reintroduces distributed training overhead that the formula does not capture.

Term three: evaluation

Routinely underestimated by an order of magnitude, because it is budgeted as “run the test set” and is actually four suites plus judgement plus human review — see the evaluation page for why the sample sizes have to be as large as they are.

one-off
  build 4 suites, ~400 examples each          engineering days
  human-label 50 pairs to validate the judge  hours

per evaluation run (and there is one per attempt)
  judge tokens = 400 examples
               x 2 orderings (position-bias control)
               x (2 responses + prompt + rubric)
               x 2 candidates (base and fine-tune)
  human review of disagreements and regressions

The doubling for position-bias control is not optional and it is frequently what pushes this term past the compute term. A judge run over four suites at four attempts is a meaningful token bill on its own.

Term four: serving, which never ends

The only recurring term, and therefore the one that dominates any horizon longer than a quarter. It is covered in full in the serving page, including the break-even calculation between dedicated capacity and per-token pricing. The short version for a budget:

• Per-token hosted fine-tune: volume × premium over the base price. Elastic, scales with usage, easy to model.
• Dedicated capacity: hourly rate × warm replicas × 730 hours, independent of traffic. Divide by your real utilisation to get the honest cost per served token.
• Re-training on base deprecation: the full one-off cost again, on someone else’s schedule. Amortise it at roughly once a year and you will not be far wrong.

A worked structure

Fill this in with your own figures. The value is in the relative sizes, which are remarkably stable even as every absolute number moves.

Two figures are worth computing before the project starts, because together they decide it. The first is the payback period: one-off costs divided by the monthly saving over the current arrangement. If that is longer than the expected life of the task — and tasks in a young product change every few months — the fine-tune does not pay back at all, whatever the quality result. The second is the cost of the alternative you are not pricing: what the same volume costs on an unmodified model today, including the prompt tokens a fine-tune would let you delete. Format tuning that removes 800 tokens of schema instruction from every request has a saving that is computable in advance and is frequently larger than the one people are actually arguing about.

If the payback is under about six months on stated assumptions, the case is usually sound. If it is over a year, the honest reading is that you are buying something other than cost — latency, data residency, control, a model nobody can deprecate — and the proposal should say so in those words rather than presenting a saving that will not arrive.