Distilling a Reasoning Model Into a Small One

The distillation that produced small reasoning models is not the logit-matching technique the name suggests. It is supervised fine-tuning on traces a large model generated — and the surprise in the published results is that this beat training the small model with reinforcement learning directly.

What is actually transferred

Classical knowledge distillation trains a student to match a teacher’s output distribution, which needs access to the teacher’s logits. Reasoning distillation does something simpler and more portable: the teacher solves a large set of problems, its full traces are kept where the final answer was correct, and the student is fine-tuned on those problem-and-trace pairs as ordinary text.

So what transfers is a behaviour pattern, not a weighted average of predictions. The student learns to decompose before answering, to check intermediate results, to notice a contradiction and back up, to state constraints before applying them. The teacher supplies demonstrations of a procedure; the student learns the procedure and applies it with its own, smaller, store of knowledge.

That framing predicts the technique’s limits accurately. It transfers the habit of reasoning well. It cannot transfer facts the student does not have room for, and it will not make a small model able to do a step whose difficulty exceeded it in the first place.

The result that made this standard

DeepSeek’s R1 report in January 2025 is the reference. Alongside the main model, they generated a large corpus of reasoning traces — the report describes on the order of 800,000 curated samples — and used it to fine-tune existing open dense models in the Qwen and Llama families, at several sizes, with no reinforcement learning stage on the students at all.

The reported outcome was that these distilled small models substantially outperformed their own base models on reasoning benchmarks, and that the effect held down to quite small parameter counts. That result is the direct cause of the wave of small reasoning-capable open models that followed: the recipe was published, the ingredients were downloadable, and the compute required was fine-tuning compute rather than reinforcement-learning compute.

Why the small model cannot just learn it

The same report contains the more interesting finding, and it is a negative one. They also ran large-scale reinforcement learning directly on a small base model — the same procedure that produced the large reasoning model — and reported that it performed worse than simply distilling from the large model. The paper draws the conclusion explicitly: distillation from a stronger reasoner is more effective and more economical than reinforcement learning on the smaller model.

The mechanism is a search problem. Outcome-reward reinforcement learning only produces signal when the model occasionally succeeds; if the small model essentially never solves a hard problem, the reward is almost always zero and there is nothing to learn from. The large model can find the solutions because it is large. Distillation hands the small model the search results instead of asking it to repeat the search — the same economy described in search as a training-time technique, where expensive exploration is paid once and amortised into weights.

Doing it for your own task

You are unlikely to reproduce a general reasoning model, and you do not need to. The version that pays for a normal engineering team is narrow: take one task you run at volume, distil a small model for that task alone, and keep the large one for the tail.

1. COLLECT   Run the teacher on 5k-50k of YOUR real inputs, not a
             public benchmark. Keep the full trace and the answer.

2. FILTER    Keep only traces whose final answer passes an automatic
             check. This is rejection sampling and it is the step that
             decides quality — an unfiltered corpus teaches the
             student the teacher's mistakes as confidently as its
             successes.

3. DEDUPE    Near-duplicate inputs produce near-duplicate traces and
             overweight whatever is common in your traffic.

4. TRAIN     Supervised fine-tune the small model on
             (input -> trace + answer). LoRA is usually enough.

5. EVALUATE  Against the teacher on a held-out set, on accuracy AND
             on cost AND on latency. The student wins on two of three
             by construction; the question is the size of the gap on
             the third.

Step two is where projects succeed or fail. Without an automatic check, you cannot filter, and an unfiltered corpus of teacher traces transfers the teacher’s errors along with its skill. Which puts this technique firmly in the verifiable-domain regime described in reasoning by domain.

Step five is where you find out whether it was worth doing at all, and it needs the comparison stated honestly. The student is cheaper and faster than the teacher by construction, so the only real question is the accuracy gap on your held-out set — and the alternative you are measuring against is not “the teacher” but “the teacher on the hard cases and the student on the rest”, which is routing. If a router captures most of the saving with none of the training, maintenance or model-hosting burden, that is the answer, and it very often is for teams below a certain volume.

Limits, and the licence question

• The domain does not generalise. A student distilled on maths traces is better at maths. It is not thereby better at your contract analysis, and evaluating it only on the distillation domain will hide that completely.
• Verbosity is inherited. Students copy trace length along with trace structure, which means a small model can end up slower per answer than a larger non-reasoning one. Measure total tokens, not parameters.
• Failure modes transfer too. A student trained on filtered-correct traces still learns the teacher’s characteristic way of being confidently wrong, because the filter was on the answer, not on the reasoning.
• Check the terms. Several commercial providers prohibit using their model outputs to train competing models. That is a contractual question about a specific provider and a specific date, not a technical one, and it is the first thing to settle before step one — the technique works regardless of whether you are permitted to use it.

The wider effect of all this is worth stating plainly, because it shapes what the model market looks like. Distillation makes capability diffuse downward quickly and cheaply once someone has paid to create it. A capability that cost a great deal of reinforcement-learning compute to discover can be transferred into a small open model for fine-tuning money, which compresses the window in which a frontier capability is exclusive. That dynamic is also, not coincidentally, the commercial reason so many providers stopped showing you the trace.