How a Neural Network Learns: Gradient Descent by Hand

Gradient descent is three lines of arithmetic repeated. The fog metaphor gets in the way, because the actual procedure is short enough to do on paper, and doing it once removes most of the mystery from everything built on top.

A model with two parameters

Fit a line ŷ = wx + b to three points: (1, 2), (2, 4), (3, 6). The answer is obviously w = 2, b = 0, which is exactly why it is a good example — you can see whether a step moved in the right direction.

The loss is mean squared error, L = (1/n) Σ (ŷᵢ − yᵢ)². Start deliberately badly, at w = 0 and b = 0. Every prediction is 0, the errors are −2, −4 and −6, and the loss is (4 + 16 + 36) / 3 = 18.67.

The two derivatives

The gradient is the vector of partial derivatives of the loss with respect to each parameter. Differentiate the loss, applying the chain rule once — the outer function is the square, the inner is ŷᵢ − yᵢ = wxᵢ + b − yᵢ:

∂L/∂w = (2/n) Σ (ŷᵢ − yᵢ) · xᵢ
∂L/∂b = (2/n) Σ (ŷᵢ − yᵢ) · 1

Read them rather than memorising them. Both are an average of the errors; the difference is that w is weighted by the input that multiplied it, and b is not, because b was multiplied by 1. That pattern — the gradient with respect to a weight is the error times the thing that weight was applied to — is the whole of backpropagation as well, repeated layer by layer.

One question is usually skipped: why move along the gradient rather than in some other direction? Because it is provably the steepest one. The change in loss for a small step in a unit direction u is the directional derivative ∇L · u, and by the Cauchy-Schwarz inequality a dot product with a fixed vector is maximised when u points along that vector. So ∇L is the direction of fastest increase, −∇L the fastest decrease, and gradient descent is not a heuristic — it is the locally optimal move for a step of a given size. The word locally is doing real work: it says nothing about the best direction over a long step, which is why the learning rate and the loss curvature are inseparable.

One step, with the arithmetic

Substituting the errors −2, −4, −6 and the inputs 1, 2, 3:

∂L/∂w = (2/3)[(−2)(1) + (−4)(2) + (−6)(3)]
      = (2/3)(−2 − 8 − 18) = (2/3)(−28) = −18.67

∂L/∂b = (2/3)[(−2) + (−4) + (−6)] = (2/3)(−12) = −8.00

Both are negative, which says the loss falls if w and b increase. Now the update. Move against the gradient, scaled by a learning rate η = 0.1:

w ← 0 − 0.1 × (−18.67) = 1.867
b ← 0 − 0.1 × (−8.00)  = 0.800

Check it did something. New predictions: 2.667, 4.533, 6.400. New errors: 0.667, 0.533, 0.400. New loss: (0.444 + 0.284 + 0.160) / 3 = 0.296. From 18.67 to 0.296 in one step, and w is already near 2. Run the same three lines again and it keeps closing.

It is worth doing the second step too, because it shows the character of the process. The new errors are all positive now — the model has overshot slightly, predicting above every target — so the gradients flip sign: ∂L/∂w = (2/3)(0.667 + 1.067 + 1.200) = 1.96 and ∂L/∂b = (2/3)(1.600) = 1.07. The update pulls both parameters back down, to w = 1.671 and b = 0.693. The trajectory is not a straight slide into the answer; it is a sequence of corrections that alternate and shrink. Every loss curve you will ever look at is that behaviour, averaged over millions of parameters.

The learning rate is not decoration. At η = 0.1 this converges; make it large enough and each step overshoots further than the last and the loss diverges to infinity, which is the single most common reason a training run produces NaN. There is a threshold, it depends on the curvature of the loss, and it can be derived exactly.

Ten lines of NumPy, if you would rather watch it than read it:

import numpy as np

x = np.array([1.0, 2.0, 3.0])
y = np.array([2.0, 4.0, 6.0])
w = b = 0.0
eta = 0.1

for step in range(50):
    err = (w * x + b) - y
    gw = 2 * np.mean(err * x)
    gb = 2 * np.mean(err)
    w -= eta * gw
    b -= eta * gb
    print(step, round(w, 4), round(b, 4), round(np.mean(err ** 2), 6))

What changes at a billion parameters

Almost nothing, and this is the point of doing it small. The gradient becomes a vector with one entry per parameter instead of two, and the update is still w ← w − η∇L applied elementwise. Three things are added by scale:

• The gradient is estimated, not computed. Averaging the loss over the whole dataset every step is unaffordable, so a random minibatch stands in for it. That makes the descent stochastic — hence SGD — and the noise in the estimate falls as 1/√B in the batch size.
• The derivatives come from backpropagation. With millions of parameters arranged in layers, computing each partial derivative separately is hopeless. Reverse-mode differentiation gets all of them in one backward sweep for roughly the cost of one forward pass.
• The raw update is usually not what is applied. Momentum, per-parameter scaling and weight decay all modify it, and AdamW is the current default. They change the step; they do not change what a gradient is.

What nobody can prove about it

For the line above, the loss surface is convex: one minimum, and gradient descent with a small enough step provably finds it. For a neural network the surface is not convex, has enormous numbers of critical points, and there is no theorem promising that SGD lands anywhere good.

It does anyway, reliably, at scale, and why is genuinely open research. Beware of any explanation that sounds settled — the honest summary is that overparameterised networks have loss landscapes with properties we can partially characterise and cannot yet fully explain, and that this gap is one of the reasons machine learning stays empirical. Every practice in this cluster — held-out splits, cross-validation, honest error bars — exists because the theory does not tell you what you are going to get.