Distillation: Teaching a Small Model From a Big One

“We distilled GPT-scale behaviour into a 7B model” almost never means what the distillation literature means. The difference is not pedantry — the two techniques have different requirements, different results and different legal exposure.

Two things called distillation

The distinction matters because the first is not available to you unless you host the teacher. Hosted APIs return sampled tokens, and at most top-k log-probabilities for a handful of candidates — not the full distribution over a 128k vocabulary, which is what logit matching needs. If you are calling an API, you are doing sequence-level distillation whatever you call it.

Soft targets, and why they carry more

The 2015 paper’s insight is about information content. A hard label says “the answer is 7”. The teacher’s distribution says “7 with probability 0.82, 1 with 0.09, 9 with 0.05, and everything else near zero” — which additionally communicates that 1 and 9 are the plausible confusions and 4 is not. That structure over the wrong answers is the part that transfers, and the paper calls it dark knowledge.

Temperature is the mechanism for exposing it. Softmax at T = 1 on a confident prediction is close to one-hot and carries little beyond the label; raising the temperature flattens the distribution so the relative ordering of the low-probability options becomes visible in the gradient. The student is trained at the same raised temperature and served at T = 1.

teacher softmax at temperature T
  p_i(T) = exp(z_i / T) / Σ_j exp(z_j / T)

student loss
  L = (1−λ) · CE(student, hard_label)
    +    λ  · T² · KL( student(T) ‖ teacher(T) )

the T² factor keeps the gradient magnitude of the soft term
comparable to the hard term as T is raised.

One consequence worth carrying: the useful signal density here is far higher per example than in sequence-level distillation. Logit distillation extracts a full distribution per position; sequence-level extracts one sampled token per position. That is why the former needs fewer examples and why the latter needs a lot of generated text.

Sequence-level distillation

The practical recipe, which is where most of this actually happens:

• Collect prompts from real traffic — the whole point is to match your distribution, and synthetic prompts drift from it.
• Generate teacher outputs. Sampling several candidates per prompt and keeping the best, by a verifier or a rubric, is meaningfully better than taking one greedy sample. This is rejection sampling and it is the single largest quality lever in the recipe.
• Filter hard. Anything that fails a schema check, a unit test, a factual verifier or a length constraint comes out. An unfiltered teacher output is a teacher error you are about to train on permanently.
• Fine-tune the student on the survivors as ordinary SFT. Everything in the SFT page applies unchanged.

Two choices inside that recipe do most of the work. The first is where the student starts: an already instruction-tuned small model needs far less data to reach a given quality than a raw base, because it already has the conversational behaviour and you are only teaching the task. The second is the sampling temperature of the teacher. Greedy decoding gives you the teacher’s modal answer every time, which produces a low-diversity corpus and a student that inherits exactly one way of solving each problem; sampling at a moderate temperature with rejection filtering gives you several valid solution paths per prompt, which is more expensive and generalises better. That trade-off — diversity bought with generation budget — is the main knob in the whole pipeline.

What the original papers reported

Two reference points, both from the authors’ own published claims rather than any run of ours.

• DistilBERT (Sanh et al., 2019, arXiv 1910.01108). Reported a model 40% smaller than BERT-base and 60% faster while retaining 97% of its GLUE language-understanding performance, using a triple loss combining language modelling, distillation and cosine embedding terms. It remains the cleanest published demonstration that most of a model’s task performance survives a large reduction in size.
• Kim & Rush, 2016. Introduced sequence-level knowledge distillation for neural machine translation and showed a student trained on teacher-generated translations outperforming one trained on the original reference translations — a striking result, and the origin of the observation that teacher output can be easier to learn than ground truth because it is more internally consistent.

A necessary counterweight: Gudibande et al. (2023, The False Promise of Imitating Proprietary LLMs, arXiv 2305.15717) trained models on outputs imitating a much stronger proprietary model and found that the students matched the teacher on stylistic and human-preference measures while showing much smaller gains on factuality and reasoning benchmarks. Their conclusion — that imitation transfers style far more readily than capability — is the most important caveat on this entire page.

A cost model, with its assumptions

Distillation is usually justified by inference savings. Whether it pays back is arithmetic with three terms, and the assumptions have to be stated because none of them are universal.

one-off
  generation  =  N examples x tokens/example x teacher price
  training    =  student GPU-hours x hourly rate
  evaluation  =  judge tokens + human review hours

recurring, per month
  saving      =  monthly volume x (teacher price − student price)

payback (months)  =  one-off / monthly saving

assumptions that must hold, or the model is wrong:
  · the student's per-token price really is lower once serving
    overhead is included (see /learn/serving-fine-tuned-models —
    dedicated capacity can erase the per-token gap entirely)
  · quality is acceptable, verified on a suite with enough
    examples to resolve the difference
  · the task does not change, because a task change means
    regenerating and retraining
  · the teacher's terms of use permit training on its outputs

The third assumption is the one that usually fails first. Distillation fixes the student at the teacher’s behaviour as of the day you generated the data, and a task that is still evolving will outrun it. Narrow, stable, high-volume tasks are where this pays; broad or moving ones are where teams end up regenerating the corpus quarterly and wondering where the saving went.

What distillation cannot transfer

• Capability the student architecture cannot express. A 1B model cannot be taught multi-step reasoning it has no capacity for. Distillation redistributes capability, it does not create it.
• Breadth. A student trained on one task’s outputs is good at that task. Distilling general capability requires a corpus with the diversity of the general case, which is a fundamentally different scale of project.
• Calibration. Students trained on teacher outputs inherit the teacher’s confidence without its basis, which tends to make them confidently wrong in exactly the places the teacher was confidently right.
• Legal permission. Several major API providers prohibit using their outputs to train competing models. This is a contract term, it varies by vendor and by date, and it is covered in the licensing page.