Backpropagation Explained Without Calculus Anxiety

Backpropagation is not an algorithm for computing derivatives. It is an ordering for computing them that reuses intermediate results, and it is the difference between training a large model and not being able to.

A network small enough to do by hand

Two weights, one hidden unit, one input, squared error. Written as a chain of four operations:

a = w1 · x          (pre-activation)
h = relu(a)         (hidden activation)
o = w2 · h          (output)
L = (o − t)²        (loss against target t)

Set x = 1, w1 = 0.5, w2 = −2, t = 1. The forward pass: a = 0.5, h = 0.5 (ReLU passes positives through), o = −1, and L = (−1 − 1)² = 4. We want ∂L/∂w1 and ∂L/∂w2.

The backward pass, with numbers

Start at the loss and walk back, carrying one running quantity: the derivative of the loss with respect to whatever you are currently standing on.

∂L/∂o  = 2(o − t)        = 2(−1 − 1)   = −4
∂L/∂w2 = (∂L/∂o) · h     = −4 × 0.5    = −2
∂L/∂h  = (∂L/∂o) · w2    = −4 × (−2)   = +8
∂L/∂a  = (∂L/∂h) · relu'(a) = 8 × 1    = +8    (a > 0, so relu' = 1)
∂L/∂w1 = (∂L/∂a) · x     = 8 × 1       = +8

Two rules produced all of that. At a multiplication, the derivative flowing back into one factor is the incoming derivative times the other factor — which is why ∂L/∂w2 is multiplied by h and ∂L/∂h by w2. At an elementwise function like ReLU, multiply by that function’s derivative, which for ReLU is 1 where the input was positive and 0 where it was not.

Notice what ∂L/∂w1 = +8 means: increasing w1 increases the loss, so gradient descent will decrease it. And notice that a negative w2 flipped the sign on the way back. That sign flip propagating through a hundred layers is where a lot of training pathology comes from.

The ReLU line also explains “dead ReLU”: had a been negative, relu’(a) = 0, the backward pass would multiply by zero, and w1 would receive no gradient at all from this example — not a small one, exactly none.

You will never write this out again, and that is the point of writing it once. A framework records the operations of the forward pass as a graph, and loss.backward() walks that graph in reverse applying exactly the two rules above at each node. Knowing what it is doing is what lets you read the failures: a gradient that is None means the graph was broken somewhere — a tensor detached, a value converted to a Python float, an operation done under a no-grad context. The parameter is not being trained, silently, and no error is raised.

Why backwards and not forwards

This is the part most explanations skip, and it is the reason the method has a name.

You could compute derivatives forwards instead: push a derivative with respect to w1 through the network alongside the values. That works, and it costs one sweep per parameter you differentiate with respect to. Going backwards costs one sweep per output you differentiate.

A trained model has one output that matters — the scalar loss — and anywhere from thousands to hundreds of billions of parameters. So forward mode costs N sweeps and reverse mode costs one. At N = 10⁹ that is not an optimisation, it is the difference between possible and impossible. The general statement, from automatic differentiation: forward mode is cheap when inputs are few, reverse mode is cheap when outputs are few, and supervised learning is the extreme case of the second.

The cost of that one backward sweep is worth knowing as a number, because it is the basis of every training-budget estimate you will see. Each weight is involved in two multiplications on the way back — one to produce the gradient with respect to the weight, one to pass the derivative on to the layer below — against one on the way forward. So a backward pass costs roughly twice a forward pass, and a full training step about three times. That is where the familiar rule of thumb comes from: training compute is approximately 6 × parameters × tokens floating-point operations, six being two per multiply-accumulate times three for forward plus backward. It is an approximation with attention and normalisation overheads folded out, and it is close enough to size a cluster.

What has to be stored

Look at the backward pass again. ∂L/∂w2 needed h, and ∂L/∂a needed the sign of a. Both are forward-pass values. So the forward pass cannot throw its intermediates away — they must be kept until the backward pass consumes them.

That is why training memory is dominated by activations rather than weights, and why it scales with batch size and depth together. Inference keeps nothing: a token goes through, the layer output is consumed by the next layer, and the memory is reused. This single asymmetry is behind several things that otherwise look unrelated:

• A model you can serve comfortably may not fit for full fine-tuning on the same hardware, because serving stores no activation graph.
• Gradient checkpointing exists to trade time for memory — discard most activations and recompute them during the backward pass, typically costing roughly one extra forward pass.
• LoRA reduces the optimiser and gradient memory by training few parameters, but the activations still have to flow through the frozen backbone, which is why it saves less than the parameter ratio suggests.

Why deep stacks used to fail

The backward pass is a product. Each layer multiplies the incoming derivative by something, so after k layers the gradient reaching the bottom is a product of k factors. If those factors average 0.9, then after 30 layers you have 0.9³⁰ ≈ 0.042, and after 100 layers 0.9¹⁰⁰ ≈ 0.000027. The early layers get essentially no signal. Average 1.1 instead and the product explodes the other way.

Every structural trick in modern deep networks is aimed at keeping that product near one. Residual connections add an identity path, so the local factor is 1 plus something rather than something. Layer normalisation keeps activation scales from drifting. Careful initialisation sets the factors near one at step zero. None of them is decoration; they are all the same fix for the same product.