Regularisation: L1, L2, Dropout and Early Stopping

Every regularisation technique trades bias for variance. The interesting question is not whether they work — the decomposition guarantees the direction — but what each one does to a weight, which is where they stop being interchangeable.

One goal, four instruments

Restrict what the model can express, so that it has fewer ways to fit noise. You pay in bias: the restricted model class no longer contains the truth as closely. When the variance saved exceeds the bias added, held-out error falls, and that is the only test that matters.

The four instruments below do this at different points: two by adding a term to the loss, one by perturbing the network during training, one by stopping the optimiser early. Their effects overlap, which is why stacking all four at maximum strength usually produces a model that underfits and a fortnight of confusion.

L2, and where the name weight decay comes from

Add half the squared norm of the weights, scaled by λ, to the loss:

L' = L + (λ/2)‖w‖²

∂L'/∂w = ∇L + λw

w ← w − η(∇L + λw)
  = w − ηλw − η∇L
  = (1 − ηλ)·w − η∇L

That last line is the whole story and it is rarely shown. Before the data term touches it, every weight is multiplied by (1 − ηλ). It decays, geometrically, every single step — which is where the name weight decay comes from, and why the two terms are used interchangeably even though one is a penalty and the other is an update rule.

Put numbers on it. With η = 10⁻³ and λ = 0.1, the factor is 0.9999 per step. Over 10,000 steps, a weight receiving no gradient at all shrinks by 0.9999¹⁰⁰⁰⁰ ≈ 0.37 — it retains about a third of its value. Over 50,000 steps, about 0.7%. Any parameter the data does not actively defend is pulled toward zero, and that is precisely the behaviour you want from a penalty.

One consequence that catches people: because the decay factor contains η, changing the learning rate silently changes the regularisation strength. That coupling is exactly what AdamW decoupled, and it is the reason the W is in the name.

Why L1 gives exact zeros and L2 does not

Now penalise the absolute value instead: L' = L + λ‖w‖₁. The derivative of |w| is sign(w), so:

L2:  pull toward zero = λ·w         proportional to the weight
L1:  pull toward zero = λ·sign(w)   the same size for every weight

Follow a weight of size 0.001 under each. Under L2 the pull is 0.001λ — vanishingly small, so the weight drifts toward zero and never arrives. Under L1 the pull is λ, the same as it would be for a weight of size 10, so a small weight is pushed straight through zero and the subgradient at zero holds it there for any gradient smaller than λ. That is sparsity, derived rather than asserted: L1 zeros a coefficient exactly when the data’s pull on it is weaker than the penalty.

Which is why L1 doubles as feature selection — the zeroed columns can be deleted — and why elastic net (both penalties at once) exists for the common case of correlated features, where pure L1 arbitrarily picks one of a correlated group and discards the rest.

Dropout, and what it is equivalent to

During training, zero each activation independently with probability p and scale the survivors by 1/(1−p) so the expected sum is unchanged. At inference, do nothing at all — the scaling already made the training-time expectation match.

The interpretation from Srivastava and colleagues (JMLR, 2014) is that each minibatch trains a different random subnetwork, so a layer of n units is training an ensemble of up to 2ⁿ thinned networks with shared weights, and inference approximates averaging them. The practical effect is that no single unit can be relied upon, so the network cannot build a brittle chain of co-adapted detectors.

Worth knowing: dropout has largely fallen out of the recipe for large transformers, which are often trained with dropout at zero. When the dataset is enormous relative to the number of passes over it, there is little to memorise and the noise costs more than it saves. It remains useful on small datasets and in fine-tuning.

Early stopping is regularisation

Watch held-out loss, keep the checkpoint where it was lowest, stop when it has not improved for a while. It feels like an operational convenience and it is a genuine penalty.

The reason: gradient descent from a small initialisation reaches large-magnitude weights only after many steps, since each step moves a weight by at most η·|∇L|. Stopping after t steps therefore bounds how far weights can have travelled, which is a constraint on their norm — the same thing L2 imposes directly. For a quadratic loss the correspondence can be made precise, with λ ≈ 1/(ηt): more steps means weaker effective regularisation. Away from the quadratic case it is an analogy rather than an identity, and it is stated here as one.

Cost and caveat: early stopping is free, and it consumes your validation set. Choosing the stopping point is a decision fitted to held-out data, so that data is no longer an unbiased estimate — which is what the third split is for.

Which one to reach for