Gradient Descent Visualiser

Drag the learning rate and watch a real optimiser walk a loss surface down, converge, crawl or diverge.

Learning rateThe one control. The stability limit below is exact for this surface — cross it and the path spirals out rather than in.

0.150

How stretched the bowl isthe ratio between the two curvatures — real loss surfaces are far worse than 60

8 : 1

Momentumfraction of the previous step carried into this one

0.00

Steps

Start x

Start y

Loss after the last step

diverged

The learning rate is above the stability limit of 0.1250 for this surface, so every step overshoots by more than it corrects.

The loss surface is L(x, y) = x² + c·y², drawn as contours. The line is the actual path the optimiser took — every point is computed, not sketched.

Steps taken: 5
Starting loss: 16.7600
Final loss: diverged
Stability limit on the learning rate: 0.1250
Your learning rate as a share of it: 120%
Steps to a loss below 0.001: not reached

What this assumes: the surface is a clean quadratic bowl, which is why the stability limit can be stated exactly: for L = ax² + by², the update multiplies each coordinate by (1 − 2·lr·curvature) plus the momentum term, so it converges only while that factor stays inside ±1 — giving lr < 1/max(a,b) without momentum and lr < (1+m)/max(a,b) with it. Real loss surfaces are not quadratic, are not convex, and the gradient you get is a noisy estimate from one mini-batch rather than the exact slope, so treat the limit as the best case. Every point on the path above is a real computed step.

Two knobs, one lesson: the largest learning rate you can use is set by the steepest direction, and the speed you actually get is set by the shallowest one. Stretch the bowl to 30 and the picture becomes unmistakable — the path bounces across the narrow valley, making almost no progress along its floor, because a rate small enough not to diverge in y is far too small to move in x. That single geometry is behind most of what optimiser design has been doing for a decade.

Momentum is the cheapest fix on this surface. It averages the oscillation in the steep direction, where consecutive gradients point opposite ways and cancel, while accumulating in the shallow direction, where they agree. Set momentum to 0.9 with a stretch of 20 and watch the bounce collapse into a curve. Per-parameter methods like Adam attack the same problem from the other side, by rescaling each direction so the bowl is closer to round.

What this leaves out is everything that makes real training hard: stochastic gradients, non-convexity, saddle points, batch-size effects and learning-rate schedules. It also has two parameters rather than a few billion. What survives the jump in scale is the relationship — if your loss diverges, the rate is above the limit set by the sharpest curvature in the model; if it plateaus, you are crawling along a flat direction and no amount of patience fixes the geometry.

Gradient Descent Visualiser

The reasoning behind this

Related tools