Layer Normalisation, Residuals and Why Deep Nets Train at All

Two lines of a transformer block look like plumbing and are load-bearing engineering: the addition that carries the input past the computation, and the normalisation next to it. Get the order of those two wrong and a deep model either needs a careful warmup schedule or does not train.

Why depth was the problem

Backpropagation through a stack of L transformations multiplies L Jacobians together. If the typical factor is a little below one, the product shrinks exponentially with depth and the early layers receive essentially no gradient. A little above one, it explodes. Neither is a bug in the optimiser; it is what happens when you multiply many numbers. Before residual connections, this is what made very deep networks untrainable in practice.

The residual, derived

A residual block computes y = x + F(x) rather than y = F(x). Differentiate:

  y = x + F(x)
  dy/dx = I + dF/dx

Across L blocks the gradient contains the product

  (I + J_L)(I + J_{L-1}) ... (I + J_1)

Expand it: one of the terms in that expansion is I times I times ... times I.
There is always a path from the loss to layer 1 that passes through NO
transformation at all.

That surviving identity term is the whole trick. The gradient can no longer be killed by a long chain of small factors, because it does not have to pass through the chain. In the forward direction the same structure means each block writes an increment to a shared residual stream rather than replacing it — the picture used throughout the layer-by-layer walkthrough.

What normalisation does

LayerNorm takes a vector, subtracts its mean, divides by its standard deviation, then applies a learned scale and shift. Note what it normalises over: the features of a single token. Not the batch. That independence from batch composition is why transformers use it — the same token gets the same treatment whether it arrives alone or with 63 others, which batch normalisation cannot promise and which matters enormously for inference.

RMSNorm (Zhang and Sennrich, 2019) drops the mean subtraction and divides by the root mean square only. Fewer operations, no measurable loss in quality in their experiments, and it is what most current open models use. Nothing else in this page changes if you substitute it.

Pre-norm vs post-norm

Two ways to combine the pieces:

post-norm (Vaswani et al., 2017):   y = LN(x + F(x))
pre-norm  (used by ~everything now): y = x + F(LN(x))

The difference looks cosmetic and is not. In pre-norm, the residual path from the embedding to the final layer is a pure sum of increments — nothing on that path is normalised, so the identity term derived above survives intact all the way down. In post-norm, the identity path passes through a LayerNorm at every single layer, and LayerNorm’s derivative rescales whatever passes through it. The clean identity is gone.

Xiong et al. (2020), “On Layer Normalization in the Transformer Architecture”, made this precise: at initialisation, the expected gradient magnitude near the output of a post-norm transformer grows with depth, while pre-norm’s does not. That is the theoretical content of a piece of practitioner folklore — post-norm transformers need a learning-rate warmup and are sensitive to how long it is; pre-norm transformers train without one. Anyone who tried to train a deep post-norm model in 2018 and watched the loss diverge in the first few hundred steps was seeing exactly this.

Pre-norm has its own cost, and it is the reason the question is not fully closed. Because every block adds to an unnormalised stream, the magnitude of the stream grows with depth, so a fixed-size increment from layer 60 is proportionally smaller than the same increment from layer 5. Later layers contribute less, which is sometimes described as a loss of effective depth. Hybrids exist: sandwich norm places a normalisation on both sides of the block, and DeepNorm (Wang et al., 2022) rescales the residual branch by a depth-dependent constant, which the authors used to train post-norm transformers of a thousand layers.

A related stabiliser has become common in large runs: normalising the queries and keys before computing attention scores, usually called QK normalisation. The failure it addresses is specific — attention logits growing large enough during training that the softmax saturates, at which point gradients through it vanish and the affected heads stop learning. Normalising the two vectors bounds the score magnitude by construction. It is a good example of the general shape of this subject: nearly every normalisation in a transformer is there to stop some quantity from growing without bound in a way that only shows up at scale.

What you can feel from outside

This is a training-time concern, so most of it is not visible to someone calling an API. Three consequences leak out anyway.

• Normalisation layers are the precision-sensitive part. Mixed-precision serving typically keeps norms and their statistics in higher precision even when weights are 8-bit or 4-bit, because a division by a badly-rounded standard deviation propagates through the whole block. It is one of the reasons two quantizations of the same model can behave differently.
• The final norm sets the scale of the logits. Which means it sets how peaked the distribution is before temperature is applied at all — the starting point for everything in sampling parameters.
• Fine-tuning inherits the sensitivity. If you are training on open weights and the loss diverges in the first hundred steps, the learning rate and the warmup are the first two things to look at, for the reason derived above.