Transfer Learning: Standing on a Bigger Model's Shoulders

Transfer learning is three different procedures that share a name, and the confusion between them is expensive. They are cleanly separated by one question: which parameters are you allowed to change?

The premise, and its evidence

A model trained on a large general task learns intermediate representations that are useful beyond that task. In a vision network, early layers respond to edges and textures — which are not specific to the thousand categories it was trained on. In a language model, the representations encode syntax and topic structure that any text task needs.

The classic empirical study is Yosinski, Clune, Bengio and Lipson (NeurIPS 2014, “How transferable are features in deep neural networks?”), which measured layer by layer how transferable representations are and reported the pattern that early layers are general and later layers grow task-specific. That result is why the standard recipe freezes the bottom and retrains the top rather than the reverse.

Three regimes, by which weights move

Feature extraction (nothing in the backbone moves)

Run the pretrained model, take an intermediate representation, and fit a small supervised model on top of it. The backbone is a fixed function, so you can run it once over your dataset, cache the vectors, and then train the head in seconds — no GPU needed for the second part, and you can try twenty heads on the cached features.

The modern version of this is embedding-based classification: send your text to an embedding model, store the vectors, fit logistic regression. It is cheap, it is auditable, and it is often competitive with fine-tuning on small label budgets.

Fine-tuning (some or all of the backbone moves)

Unfreeze part of the network and continue training on your data at a much lower learning rate. Costs a full backward pass and the optimiser state that goes with it. Buys the ability to change the representation itself, which matters when your domain is genuinely unlike the pretraining distribution.

Continued pretraining (everything moves, same objective)

Keep the original training objective and run it on a new corpus — medical text, a codebase, a language the model saw little of. This is the right tool when the gap is vocabulary and domain knowledge rather than task format, and it needs far more data than either of the other two.

Why a hundred examples can be enough

Count parameters, because it explains the sample sizes that otherwise look impossible.

training from scratch:   ~10⁸ parameters to fit
linear head on 768-dim frozen features, 5 classes:
    768 × 5 weights + 5 biases  =  3,845 parameters

The sample size you need is governed by the parameters you are fitting, not by the size of the model producing the features. Fitting 3,845 numbers from a few hundred labelled examples is an ordinary statistics problem; fitting 10⁸ from the same data is not a problem at all, it is impossible. The backbone did the hard part, on somebody else’s compute budget, and it is a fixed function that contributes no variance to your fit.

When frozen features are not enough, unfreeze gradually rather than all at once. The recipe follows from the Yosinski result: train the new head first with everything else frozen, so that a randomly initialised head does not fire large, meaningless gradients into a carefully trained backbone on step one; then unfreeze the top few blocks; then, if it is still needed, the rest. Discriminative learning rates express the same idea continuously — a lower rate for early layers, which encode general structure worth preserving, and a higher one for later layers, which are the task-specific part you actually intend to change.

Practical corollary: start with the frozen version. It gives you a number in an afternoon, it establishes the baseline any fine-tune has to beat, and if it is good enough you have avoided owning a training pipeline.

LoRA is the same trick, scaled

Low-rank adaptation applies the identical logic inside the network. To adapt a weight matrix W of shape d × k, freeze it and learn a low-rank update ΔW = B·A, where A is r × k and B is d × r:

d = k = 4,096      full matrix:  4,096 × 4,096 = 16,777,216 parameters
r = 8              LoRA update:  8 × (4,096 + 4,096) =  65,536 parameters

ratio = 16,777,216 / 65,536 = 256×  fewer trained parameters

Same principle as the linear probe — fit few parameters on top of a frozen function — applied to every attention projection rather than to the final layer. And by the optimiser-state arithmetic, trained parameters are what m and v are stored for, so the memory saving tracks that ratio. The activations still flow through the frozen backbone, so the saving is real but not 256× overall — the detail is worth reading before budgeting.

When transfer hurts

• Negative transfer. If the source domain is unrelated, the pretrained representation can be worse than a randomly initialised one, because you also inherit its biases about what is worth encoding. Rare with very general pretraining, real with narrow source tasks.
• Forgetting. Fine-tuning moves weights that supported abilities you never tested. The model gets better at your task and worse at things nobody wrote an eval for — which is why the held-out set must include the old behaviour.
• Inherited flaws. Biases, contamination and safety behaviour in the base model transfer along with the useful parts. The provenance of the backbone becomes part of your system’s provenance.
• Licence and lineage. A fine-tune inherits the base model’s licence terms. This is a legal fact about the artefact you ship, and it is checked before training, not after.