Derivatives and the Chain Rule, for People Who Forgot
10 min read · updated August 4, 2026
The chain rule says that when one thing depends on another which depends on a third, the rates multiply. That is the whole of backpropagation. This page derives one gradient by hand, checks it by nudging the input and measuring, and then does the same for every weight in a small network.
A derivative is a rate, and you can measure it
The derivative of f at x is how much f changes per unit change in x, in the limit of a tiny change. You do not have to take the limit to check an answer — a small nudge is close enough to catch a mistake, and this is the single most useful debugging technique in all of gradient-based learning.
f(x) = x^2, claimed derivative 2x, at x = 3 that is 6 Check by nudging: f(3.0000) = 9.000000 f(3.0001) = 9.00060001 (9.00060001 - 9.000000) / 0.0001 = 6.0001 6.0001 against a claimed 6. Correct.
Three derivatives cover almost everything in a neural network:
d/dx (x^2) = 2x d/dx (a * x) = a d/dx relu(x) = 1 if x > 0 else 0
The chain rule on one composed function
f(x) = (3x + 1)^2
Split it: u = 3x + 1
f = u^2
df/du = 2u
du/dx = 3
df/dx = df/du * du/dx = 2u * 3 = 6u = 6(3x + 1)
At x = 2: u = 7, df/dx = 42Check it the same way:
f(2.000) = (7.000)^2 = 49.000000 f(2.001) = (7.003)^2 = 49.042009 (49.042009 - 49.000000) / 0.001 = 42.009 42.009 against a claimed 42. Correct.
The structure is worth naming, because it is exactly what a deep network is: a chain of functions, and the gradient at the bottom is a product of the local rates all the way up. Ten layers means ten factors multiplied together, which is why gradients vanish when those factors are small and explode when they are large, and why normalisation layers and residual connections exist to keep the factors near 1.
A two-layer network, forward
One input, one hidden unit, one output. Scalars throughout, so nothing is hidden inside a matrix.
Parameters: w1 = 0.5 b1 = 0.0 w2 = 2.0 b2 = 0.0 Input: x = 1.0 Target: y = 0.5 Loss: L = (yhat - y)^2 Forward: z1 = w1 * x + b1 = 0.5 * 1.0 + 0.0 = 0.5 a1 = relu(z1) = 0.5 z2 = w2 * a1 + b2 = 2.0 * 0.5 = 1.0 yhat = z2 = 1.0 L = (1.0 - 0.5)^2 = 0.25
The same network, backward
Work from the loss down, multiplying local derivatives. Each line reuses the line above it, which is the only reason backpropagation is cheap.
dL/dyhat = 2 * (yhat - y) = 2 * (1.0 - 0.5) = 1.0
dL/db2 = dL/dyhat * dyhat/db2 = 1.0 * 1 = 1.0
dL/dw2 = dL/dyhat * dyhat/dw2 = 1.0 * a1 = 0.5
dL/da1 = dL/dyhat * dyhat/da1 = 1.0 * w2 = 2.0
dL/dz1 = dL/da1 * da1/dz1 = 2.0 * 1 = 2.0
(relu, z1 = 0.5 > 0)
dL/db1 = dL/dz1 * dz1/db1 = 2.0 * 1 = 2.0
dL/dw1 = dL/dz1 * dz1/dw1 = 2.0 * x = 2.0Check the deepest one, dL/dw1 = 2.0, by nudging:
w1 = 0.5001 z1 = 0.5001 a1 = 0.5001 z2 = 2.0 * 0.5001 = 1.0002 L = (1.0002 - 0.5)^2 = 0.5002^2 = 0.25020004 (0.25020004 - 0.25) / 0.0001 = 2.0004 2.0004 against a claimed 2.0. Correct.
Two structural points fall out of that table. First, computing the gradient for every parameter took one pass, because each step reuses the accumulated dL/d(something) from the step after it. Doing it the other way — a separate nudge-and-measure per parameter — would cost one forward pass per parameter, which for a 7B model would be seven billion forward passes per training step.
Second, the backward pass costs roughly twice the forward pass, because at each layer it computes two things: the gradient with respect to the layer’s input (to pass down) and the gradient with respect to the layer’s weights (to keep). That is where the 6 x params x tokens figure for training comes from: 2 for the forward, 4 for the backward.
Taking a step, and checking the loss went down
Gradient descent subtracts a fraction of each gradient from each parameter. With a learning rate of 0.1:
w1 <- 0.5 - 0.1 * 2.0 = 0.30 b1 <- 0.0 - 0.1 * 2.0 = -0.20 w2 <- 2.0 - 0.1 * 0.5 = 1.95 b2 <- 0.0 - 0.1 * 1.0 = -0.10 Forward again: z1 = 0.30 * 1.0 - 0.20 = 0.10 a1 = relu(0.10) = 0.10 z2 = 1.95 * 0.10 - 0.10 = 0.095 L = (0.095 - 0.5)^2 = 0.164025 Loss: 0.250000 -> 0.164025. Down by 34%.
That is a complete training step. Everything a real training loop adds — momentum, adaptive learning rates, weight decay, learning-rate schedules — modifies how the update is computed from the gradient. None of it changes how the gradient itself is obtained, and the arithmetic above is exactly what gradient descent does at every scale.
w1 would go to -1.5, the ReLU would output 0, and that unit would produce a zero gradient forever — a dead ReLU, from one step that was too big.Why depth makes gradients vanish or explode
The chain rule multiplies one factor per layer. Multiply 32 numbers together and the result is almost never near 1 — it is near zero or enormous, and which one depends on whether the typical factor is below or above 1.
Gradient reaching the first layer of a 32-layer stack, if each layer contributes a factor f: f = 0.50 -> 0.50^32 = 2.33e-10 f = 0.90 -> 0.90^32 = 3.43e-02 f = 0.99 -> 0.99^32 = 7.25e-01 f = 1.00 -> 1.00^32 = 1.00 f = 1.01 -> 1.01^32 = 1.38 f = 1.10 -> 1.10^32 = 2.11e+01 f = 1.50 -> 1.50^32 = 4.31e+05 f = 2.00 -> 2.00^32 = 4.29e+09
The window in which a 32-layer network trains at all is roughly f between 0.99 and 1.01. Outside it, either the early layers receive nothing and never learn, or the update is astronomically large and the weights leave the region where the model produces finite numbers.
Both failure modes are worse in low precision. At f = 0.5 the gradient reaching layer 1 is 2.33e-10, which is below fp16’s smallest subnormal of 5.96e-8 and becomes exactly zero — the layer stops learning with nothing in any log to say so.
Residual connections are the structural fix, and the reason is one derivative. A residual block computes y = x + f(x), so:
dy/dx = 1 + df/dx The 1 is a path with a factor of exactly 1, at every layer. Without residuals, if each block contributes 0.1: 0.1^32 = 1e-32 -> nothing reaches the bottom With residuals, the same block gives 1 + 0.1 = 1.1: 1.1^32 = 21.1 -> bounded, and trainable
The residual stream is, in gradient terms, a road from the loss straight to every layer that does not get multiplied away. Normalisation layers do the complementary job of keeping df/dx near a sensible scale so the factors stay near 1, which is why the two appear together in every transformer block.
The remaining guard is gradient clipping, which handles the explosive direction directly rather than structurally: compute the global norm of all gradients, and if it exceeds a threshold, rescale the whole vector down to that threshold.
global_norm = sqrt( sum over all parameters of g^2 )
if global_norm > max_norm:
scale = max_norm / global_norm
g <- g * scale
With max_norm = 1.0 and a spike of global_norm = 50:
scale = 0.02, and every gradient is cut 50-fold.
The direction of the update is unchanged; only its
length is. That is the point: a bad batch should not
be able to move the weights further than a good one.What changes at 7 billion parameters
- Scalars become matrices, and nothing else.
dz/dw = xbecomes an outer product;dL/da = wbecomes a multiplication by the transposed weight matrix. The chain rule is identical, applied to the same matmuls the forward pass used. - The activations must be kept. Every backward step above used a value from the forward pass —
a1,x, the sign ofz1. At scale that is why training memory is dominated by stored activations rather than weights, and why gradient checkpointing exists: throw the activations away and recompute them during the backward pass, trading about 30% more compute for a large memory saving. - The loss is cross entropy, and its gradient is simpler than this one. Softmax followed by cross entropy has the gradient
p - ywith respect to the logits — one subtraction per logit, with no exponentials in the backward pass at all. - Finite differences still work, and are still the right test. Every autograd framework ships a gradient checker that does exactly the nudge-and-measure above. If you write a custom layer, run it before you trust the layer, on a small input in fp64.