Should You Fine-Tune? A Decision Tree

Fine-tuning is a weight update. Everything below it on this ladder is a text change, a configuration change or a one-line model swap — and all three can be undone in an afternoon. That asymmetry, not any claim about quality, is why fine-tuning goes last.

The ladder, in order

Each rung is strictly more expensive to build, more expensive to maintain and harder to reverse than the one before it. Climb in order and stop at the first rung that clears your bar.

• 1. Write the instruction properly. Most “the model cannot do this” reports are an instruction that never stated the constraint, or stated it once in the middle of 900 tokens of preamble. Cost: an hour.
• 2. Few-shot with real examples. Three to eight examples of the exact input and exact desired output, drawn from your own data. This is the highest-yield rung and the one people skip fastest, because it feels like cheating. Cost: an afternoon, plus a permanent tax on every prompt’s input tokens.
• 3. Give it the facts it is missing. Retrieval, a tool call, a database lookup, a bigger slice of the document. If the model is wrong because it does not know something, this is the only rung that fixes it. Cost: days to weeks, and a second system to operate.
• 4. Try a stronger model. A frontier model on the same prompt is the cheapest capability increase available and takes about as long as editing a string. It is also the rung that most often makes rung five unnecessary.
• 5. Fine-tune. Cost: weeks, mostly on data and evaluation rather than training, plus a serving arrangement you now own forever.

The test that picks the rung

There is one diagnostic worth running before any of this, and it takes twenty minutes. Take twenty failing examples. For each one, paste the input into a chat window with the ideal output written out by hand, and ask: could a competent human, given only what is in this prompt, have produced the right answer?

• No, the information is not there — you have a knowledge problem. Rung three. No amount of training on your own logs invents a fact that was never in the prompt.
• Yes, but only after re-reading the instruction three times — you have a prompt problem. Rung one or two.
• Yes, easily, and the model still got it wrong — now you have a capability or a behaviour problem, and rungs four and five are both live options.

The third bucket is the only one where fine-tuning is even a candidate, and in practice it is usually much smaller than teams expect. Run the test before writing a data pipeline, not after.

What each rung costs

The compute bill for a small fine-tune is genuinely modest — a LoRA run on a few thousand examples is hours on one GPU. That is why the compute number is the wrong one to plan against. The costs that dominate are the ones that recur.

That last cell is the one that surprises people. A fine-tuned model is pinned to its base checkpoint. When the base is retired — and open and hosted bases alike get retired — your adapter does not transfer, and you redo the run against a new base with a new tokeniser and possibly a new chat template.

Four cases where the answer is yes

• A rigid output format at volume. Fine-tuning teaches form extremely reliably, and a format-tuned small model can hold a schema that a much larger model needs 800 tokens of instruction to hold. The saving compounds per request.
• A tone or register you cannot describe. If your style guide is easier to demonstrate than to write down — house voice, a clinical register, a specific translation idiom — examples are the right encoding and the prompt is the wrong one.
• Cost or latency at high, stable volume. Distilling a frontier model’s behaviour on one narrow task into a small model is a real and repeatable win, provided the task is narrow and the volume justifies dedicated serving.
• A classification or extraction task with a fixed label set. These respond to a few thousand labelled examples about as well as anything in this field responds to anything.

Three where it is not

• To install facts. This is the single most common reason teams fine-tune and the one with the worst evidence behind it. See /learn/format-vs-knowledge-tuning for what the literature actually found.
• To fix hallucination in general. Training on correct answers teaches the model that confident answers are what you want. If the failure is fabrication under uncertainty, the fix is grounding, not gradients.
• Because the task is “domain-specific”. Domain specificity is not itself a reason. Legal, medical and financial text are all well represented in pretraining corpora. The question is still whether the failure is knowledge, prompt or behaviour.

The tree, compressed

Is the needed information in the prompt?
├─ no  ──────────────► retrieval / tools          (rung 3)
└─ yes
   └─ Would a careful human get it right from this prompt?
      ├─ only just ──► rewrite prompt, add few-shot (rungs 1–2)
      └─ easily
         └─ Does a stronger model get it right?
            ├─ yes ──► use it; revisit if the price hurts (rung 4)
            └─ no
               └─ Can you produce 1,000+ examples of the
                  behaviour you want, and an eval that
                  proves it improved?
                  ├─ no  ─► you are not ready to fine-tune
                  └─ yes ─► fine-tune                (rung 5)

The final gate is the honest one. If you cannot produce the evaluation that would tell you the fine-tune worked, you cannot tell whether it worked, and a fine-tune you cannot evaluate is a fine-tune you will ship on vibes and quietly roll back four months later.