Fine-Tuning Open Models: The Full Pipeline

Fine-tuning is the most over-recommended technique in applied language modelling and one of the most useful when it is the right one. The difference is entirely in what you are asking it to change.

Whether to fine-tune at all

Fine-tuning teaches behaviour: a format, a register, a decision policy, a domain’s idiom, a structure the model keeps drifting away from. It is poor at teaching facts, because facts change and a model is an awkward place to store them.

• Prompt engineering first. If a longer system prompt with three examples fixes it, you are done, and you can change your mind tomorrow.
• Retrieval for knowledge. Anything that must be current, cited or auditable belongs in the context, not the weights.
• Fine-tune for consistency, style and compression. When the behaviour you want takes 2,000 tokens of instructions to describe and still slips, training it in is both more reliable and cheaper per call.
• Fine-tune to move down a size class. The strongest economic case: a tuned small model matching a large general one on one narrow task, at a fraction of the serving cost.

Before any of it, check the base model’s licence. Some licences impose naming or attribution requirements on derivatives, some restrict using outputs to train other models, and a fine-tune cannot grant permissions its base withheld.

What training costs in memory

This decides your entire approach, and it is arithmetic like everything else in this cluster. Full fine-tuning holds, per parameter: the weights, the gradients, and the optimiser state, which for Adam is two moments plus commonly a high-precision copy of the weights.

full fine-tune, Adam, mixed precision
  ~16 bytes per parameter, before activations
  8B model -> 8e9 * 16 = 128 GB   (a multi-GPU job)

LoRA on a 16-bit base
  base weights frozen        8e9 * 2  = 16 GB
  adapters + their optimiser          < 1 GB
  activations                          workload dependent

QLoRA: LoRA on a 4-bit quantised base
  base weights               8e9 * 0.5 = 4 GB
  adapters + optimiser + activations   a few GB
  -> an 8B tune fits on a single 24 GB card

That gap is why almost every fine-tune outside a large lab is a LoRA. Low-rank adaptation freezes the base and trains a pair of small matrices alongside chosen weight matrices, so the trainable parameter count drops by orders of magnitude and the optimiser state drops with it. For the behavioural changes described above, it is generally close to indistinguishable from a full tune.

The data is the project

Expect to spend most of the time here, and be suspicious of any plan that does not. A few hundred excellent examples routinely beat tens of thousands of mediocre ones, because the model is learning the distribution you show it including its inconsistencies.

• Match the serving format exactly. Same system prompt, same message roles, same chat template as production. A model tuned on one framing and served under another has been trained for a job it will not be given.
• Be ruthless about consistency. If half your examples use one JSON key ordering and half another, you have taught the model to be inconsistent. Contradictions in the data become variance in the output.
• Hold out properly, and by group. Split before you look. If several examples derive from one document or one customer, they must land on the same side of the split or your evaluation is measuring leakage.
• Include the boundaries. Examples of refusing, asking for clarification, and handling malformed input. A dataset of only successful cases produces a model that never declines.
• Mind the provenance. Outputs from another model may be restricted by that model’s terms. This catches people late, and it is a licence question, not a technical one.

LoRA and the knobs that matter

# a representative configuration; the framework varies, the knobs do not
base_model: org/model-8b-instruct
load_in_4bit: true          # QLoRA

adapter: lora
lora_r: 16                  # rank: capacity of the adaptation
lora_alpha: 32              # scaling, commonly 2x rank
lora_dropout: 0.05
lora_target_modules:        # attention projections at minimum;
  - q_proj                  # adding the MLP projections raises
  - k_proj                  # capacity and memory together
  - v_proj
  - o_proj

sequence_len: 4096
micro_batch_size: 2
gradient_accumulation_steps: 8    # effective batch = 16
num_epochs: 2
learning_rate: 0.0002             # 1e-4 to 2e-4 is the usual band
lr_scheduler: cosine
warmup_ratio: 0.03
val_set_size: 0.05

• Rank sets how much the adapter can change. 8 to 16 covers most style and format work; raise it for genuinely new capability, and expect diminishing returns quickly.
• Target modules matter more than rank in practice. Attention-only is the cheap default; including the MLP projections helps on harder adaptations at a real memory cost.
• Epochs. One to three. This is where overfitting lives — watch validation loss, and stop when it turns up rather than when the schedule ends.
• Learning rate for LoRA runs an order of magnitude above full fine-tuning rates. Divergence early usually means it is too high; nothing changing usually means it is too low or the rank is too small.

Evaluating a tune honestly

Validation loss tells you the training worked. It does not tell you the model got better at the job, and the two come apart routinely.

• Compare against the untuned base on the same held-out set with the same prompts and the same sampling settings. Without the paired baseline you have a number with nothing to compare it to.
• Test for regression outside the target task. Narrow tuning degrades general ability — instruction following, refusals, multi-turn coherence. Keep a small general set and run it every time.
• Check the mechanical properties first. Schema validity, required fields, length distribution. These usually improve dramatically and they are the reason you did this.
• Be aware you can win the eval and lose the deployment if the eval set came from the same narrow slice as the training data. Sample the held-out set from a different week.

Serving the result

Two options, and the choice has operational consequences.

Serve the adapter separately. Batching servers can load a base model once and attach multiple LoRA adapters, routing per request. This is what you want when you have several tunes, because one copy of the base serves all of them:

vllm serve org/model-8b-instruct \
  --enable-lora \
  --lora-modules support=./out/support-lora triage=./out/triage-lora
# then send model: "support" or model: "triage" per request

Or merge and ship one artefact. Merging folds the adapter into the weights, giving a normal checkpoint with no runtime adapter support required — which is what you need for local distribution:

from peft import AutoPeftModelForCausalLM
m = AutoPeftModelForCausalLM.from_pretrained("./out/support-lora")
m.merge_and_unload().save_pretrained("./merged", safe_serialization=True)

# then, for llama.cpp deployment:
python convert_hf_to_gguf.py ./merged --outfile support.gguf --outtype f16
llama-quantize support.gguf support-Q4_K_M.gguf Q4_K_M

One caution on merging: if you trained with a 4-bit base, merge into the 16-bit base rather than the quantised one, then quantise the merged result. Merging into a quantised base compounds two approximations and the output can be worse than either step suggested. Re-run the evaluation on the exact artefact you intend to serve — quantisation after merging is a change to the model, and the only version whose quality you know is the one you tested.