Supervised Fine-Tuning, Step by Step

Supervised fine-tuning is next-token prediction on examples you chose. There is no new objective, no new loss, nothing exotic — which means every hyperparameter has a mechanical justification, and you should be able to give it.

What is actually being optimised

The loss is cross-entropy over the tokens of the desired completion, conditioned on the prompt. Identical to pretraining except for two things: the data is yours, and the prompt tokens are masked out of the loss so the model is only graded on what it should generate.

That framing settles several arguments before they start. SFT can only ever increase the probability of the completions you showed it relative to alternatives. It has no notion of a bad answer — there are no negatives — so it cannot be taught what not to do except by showing it the right thing instead. Wanting the model to prefer A over B is a preference-tuning problem, not an SFT one.

It also explains why data quality dominates everything else on this page. The objective is “imitate this”. Every inconsistency in your targets is a contradiction in the gradient, and averaging two contradictory demonstrations produces a model that does neither well.

One more thing follows from the objective and it is the most useful debugging heuristic in this cluster: the model is learning the distribution of your targets, including the properties you did not intend to specify. If every target in your dataset is between 80 and 120 words, you have trained a length constraint. If every one begins with a restatement of the question, you have trained an opening formula. If none of them ever refuses, you have trained away refusal. None of these appear in your dataset specification; all of them appear in the fine-tuned model. Before a run, sample fifty targets and ask what they have in common that you did not choose.

An annotated configuration

A LoRA SFT setup for a 7B-class base on a few thousand instruction-response pairs. Every line has a reason after it; these are starting points chosen so that the first thing you change is the dataset, not the config.

# --- adapter ---------------------------------------------------------
r                    = 16        # capacity. 8 for pure format work, 32+
                                 # if targeting all linear layers
lora_alpha           = 32        # convention alpha = 2r; the update is
                                 # scaled by alpha/r, so changing r alone
                                 # changes the effective step size too
lora_dropout         = 0.05      # small datasets overfit; 0.0 above ~50k
target_modules       = ["q_proj","k_proj","v_proj","o_proj",
                        "gate_proj","up_proj","down_proj"]
                                 # covering the MLP blocks usually helps
                                 # more per parameter than raising r

# --- optimisation ----------------------------------------------------
learning_rate        = 2e-4      # LoRA scale. Full fine-tuning: 1e-5–2e-5
lr_scheduler_type    = "cosine"  # decay to ~0 by the end so the last
                                 # steps are refinement, not exploration
warmup_ratio         = 0.03      # Adam's variance estimates are garbage
                                 # for the first few dozen steps
num_train_epochs     = 2         # 1–3. Beyond 3 you are memorising
weight_decay         = 0.0       # on adapters; the base is frozen anyway
max_grad_norm        = 1.0       # one malformed example should not be
                                 # able to move the weights arbitrarily
optim                = "adamw_torch"

# --- throughput ------------------------------------------------------
per_device_batch     = 4
gradient_accumulation= 8         # effective batch = 4 x 8 = 32
gradient_checkpointing = True    # trades ~30% step time for a large
                                 # reduction in activation memory
bf16                 = True      # NOT fp16 on a bf16-trained base
max_seq_length       = 2048      # cap it; the tail of the length
                                 # distribution is what OOMs you
packing              = False     # True only if examples are independent
group_by_length      = True      # batch similar lengths, less padding

# --- correctness -----------------------------------------------------
train_on_completions_only = True # mask prompt tokens out of the loss
seed                 = 0         # so the ablation runs are comparable

Why the learning rate is ten times higher

2e-4 would destroy a full fine-tune and is unremarkable for LoRA. The reason is in the parameterisation, not in a convention.

A LoRA update reaches the effective weight through a product of two matrices, B·A, and B starts at exactly zero. At step 0 the gradient with respect to A is therefore zero as well — it is multiplied by B — so nothing happens through that path until B has moved away from the origin. The update also passes through a rank-8 or rank-16 bottleneck and is then scaled by α/r. Small steps on the factors produce much smaller steps on the effective weight, so the factors need larger steps.

The practical consequence: learning rate and rank are not independent knobs. Doubling r while holding lora_alpha fixed halves the scaling factor and quietly halves your effective learning rate, which is why a rank increase sometimes appears to make results worse. Hold α/r constant when you sweep rank, or sweep the learning rate alongside it.

Effective batch size and packing

Effective batch size is per_device_batch × gradient_accumulation × num_devices, and it is the number that matters — gradient accumulation is mathematically a larger batch, just computed over several passes. Choose the effective size for the optimisation and the per-device size for the memory you have.

For instruction tuning on a few thousand examples, an effective batch of 16 to 64 is the usual range. Smaller and the gradient is noisy enough that the loss curve is unreadable; larger and a small dataset gives you too few optimiser steps to converge — 2,000 examples at an effective batch of 128 is only about 16 steps per epoch, which is not enough for a cosine schedule to mean anything.

Packing concatenates several short examples into one sequence to avoid wasting compute on padding. It is a genuine throughput win on short data and a correctness hazard: unless the implementation resets attention at document boundaries, tokens can attend across examples and the model learns to condition on unrelated preceding text. Leave it off until throughput is the actual problem, then verify boundary handling before turning it on.

max_seq_length deserves a decision rather than a default. Set it from the 95th or 99th percentile of your token-length distribution, not from the maximum and not from the model’s context window. Setting it at the maximum means every batch is sized for an example that occurs once, which costs memory on every step for no benefit; setting it too low silently truncates, and a truncated target teaches the model to stop mid-sentence in exactly the way described in the dataset page. Compute the histogram first, choose the cap, and log how many examples it drops so the number is a decision on the record rather than a surprise.

Reading the loss curve

The order to change things in

When a run underperforms, the temptation is to sweep hyperparameters, because a sweep is easy to launch and feels like progress. The expected value is poor. In descending order of what actually moves held-out quality:

• Fix the data. Inconsistent targets, near-duplicates, truncated completions, wrong chat template. This is where the wins are and it is not close.
• Add different data, covering input shapes the dataset does not contain.
• Change epochs. The one hyperparameter with a large effect on a small dataset.
• Change target modules, then rank, then learning rate. In that order, and one at a time, with the seed held fixed.