Serving a Fine-Tuned Model: The Part Nobody Budgets For

The training run is the cheap part and it ends. Serving is the expensive part and it does not. The number that decides whether your fine-tune saves money is not the training bill, it is your requests per hour.

The billing model changes underneath you

Calling a shared, hosted model is per-token billing: you pay for what you use, the provider amortises the hardware across thousands of tenants, and an idle hour costs nothing. That amortisation is only possible because everyone is hitting the same weights.

Your fine-tuned model has different weights, so it cannot join that pool as an ordinary tenant. There are three ways it gets served, and they have different cost shapes:

The third is where budgets die, because the cost is decoupled from usage in the one direction that hurts. Doubling traffic on dedicated capacity is free until you saturate; halving traffic saves nothing.

The break-even calculation

The comparison is arithmetic. Plug in the numbers your provider quotes today rather than any figure written on a page months ago — GPU hourly rates and per-token prices have both moved substantially and in different directions.

Let
  H  = hourly cost of the dedicated deployment   (currency/hour)
  R  = replicas you must keep warm               (>= 1)
  P  = per-token price of the shared alternative (currency per 1M tokens)
  T  = your monthly token volume                 (millions of tokens)

dedicated monthly cost   =  H · R · 730
shared monthly cost      =  P · T

break-even volume
  T*  =  (H · R · 730) / P     millions of tokens per month

worked, with placeholder values you must replace
  H = 2.00 /hour,  R = 2 replicas,  P = 0.40 per 1M tokens
  T* = (2.00 · 2 · 730) / 0.40 = 7,300 M tokens/month
     = 7.3 billion tokens/month
     ≈ 2,800 tokens/second, sustained, 24/7

That last line is the one to internalise. Under those placeholder assumptions you need thousands of tokens per second, continuously, for dedicated capacity to be the cheaper option on compute alone. Substitute your own H and P and the number will move, but the shape does not: dedicated capacity is a high-volume arrangement, and most fine-tunes are deployed at volumes far below their break-even.

Which does not automatically make it wrong. Dedicated capacity buys latency predictability, data residency, freedom from a shared queue, and a model nobody can deprecate under you. Those can be worth paying for. They are just not savings, and they should not be presented internally as savings.

Why utilisation decides everything

Real traffic is not flat. If your load follows office hours, your peak might be ten times your trough, and you have three options — all of which cost something.

• Provision for peak. Simple, and you pay peak rates during the trough. At a 10:1 peak-to-trough ratio the effective utilisation can easily land in the teens, which multiplies your real cost per token by roughly the reciprocal.
• Autoscale. Better economics, but a cold start on a large model means loading tens of gigabytes of weights, and the first request after a scale-up event waits for it. Anything user-facing needs a warm floor, which is the R in the formula above and the reason it is rarely 1.
• Scale to zero. Only viable for batch and internal workloads where a multi-minute first-request latency is acceptable. When it is acceptable, it is the largest saving available.

A useful sanity check before committing: compute your average utilisation from a week of request logs, then divide your quoted hourly cost by it. That is your real cost per served token, and it is frequently several times the headline number.

Merged weights or adapters

If you trained with LoRA you have a choice at serving time, and it is not a small one.

Merged gives you an ordinary model file. No adapter runtime, no per-token overhead, works on any inference server. The cost is that one GPU serves one fine-tune: ten customer-specific models means ten deployments.

Unmerged keeps the adapter separate and applies it at runtime. There is a small per-token cost — two extra low-rank matmuls per adapted layer — and in exchange one base model in memory serves many adapters, with each adapter costing single-digit megabytes rather than a whole deployment. For anything with more than one fine-tune this is decisively the better arrangement.

The caveat from the LoRA page applies here: if you trained against a quantised base, merging is lossy, and the merged artefact is not the one your evaluation blessed. Whichever you choose, evaluate the artefact you are going to serve.

The costs that are not compute

• The base model gets deprecated. Every fine-tune is pinned to a checkpoint. When the base is retired you retrain, and re-run the whole evaluation, on somebody else’s schedule. Budget this as a recurring engineering cost, not a one-off.
• You now operate a model. Version pinning, rollback, an artefact registry, a health check that catches a corrupted adapter load, and a way to answer “which weights produced this output” three months later.
• Your evaluation suite is now infrastructure. It has to run on every candidate and be maintained as behaviour changes.
• You lose the free upgrade. Shared hosted models improve underneath you. A pinned fine-tune does not, and after a year the gap between your specialised small model and the current general one is often the wrong way round.