Learning Rate: The One Hyperparameter That Matters Most

Every guide says the learning rate is the most important hyperparameter and then gives you a range to try. The range is a consequence of something you can derive in five lines, and deriving it tells you what each failure looks like before you see it.

What the number does

The update is w ← w − η∇L. The gradient has units of loss per unit of parameter; the update has to have units of parameter. So η is the conversion factor, and it is the only hyperparameter whose job is to translate between two spaces. Everything else about training tolerates being roughly right. This one has a threshold above which nothing works at all.

Deriving the stability bound

Assumptions, stated up front: one parameter, exact gradients (no minibatch noise), and a quadratic loss L(w) = ½·k·w² with curvature k > 0 and minimum at zero. Every real loss looks like this locally near a minimum, which is why the result transfers.

∇L = k·w

w_{t+1} = w_t − η·k·w_t = (1 − ηk)·w_t

so after t steps:   w_t = (1 − ηk)^t · w_0

converges to 0  ⟺  |1 − ηk| < 1  ⟺  0 < η < 2/k

Four regimes fall straight out of that single factor (1 − ηk):

• η < 1/k — the factor is between 0 and 1. Smooth monotone convergence, slower the smaller η is. This is the “too small, training crawls” regime.
• η = 1/k — the factor is exactly 0. One step lands on the minimum. The optimum, and unknowable in advance because k is unknown.
• 1/k < η < 2/k — the factor is between −1 and 0. The weight overshoots and lands on the other side, closer each time: oscillating convergence. Loss curves that zigzag downward are here.
• η > 2/k — the factor has magnitude above 1. Each step overshoots further than the last, geometrically. Loss goes to infinity and then to NaN, usually within a few dozen steps.

So the whole usable range spans a factor of two in η, for a given curvature. That is why the learning rate is fiddly, and why the symptom of “too high” is not slightly worse results but total failure.

Why many dimensions make it worse

Drop the one-parameter assumption. In many dimensions the loss has a different curvature along each direction — the eigenvalues of the Hessian, from k_max down to k_min. The same derivation applies independently along each, so:

• Stability is set by the largest curvature: η < 2/k_max, or the steepest direction diverges and takes everything with it.
• Progress along the flattest direction goes as (1 − η·k_min)^t, which with η capped by k_max means the number of steps needed scales with the condition number k_max / k_min.

A poorly conditioned loss therefore forces you to choose between diverging and waiting. This is the exact problem that per-parameter scaling solves: Adam divides each parameter’s step by the running magnitude of its own gradients, which is a crude preconditioner and is why Adam tolerates a badly scaled problem where plain SGD needs the learning rate tuned to the worst direction.

Schedules and warmup

Since the ideal η is 1/k and the local curvature changes as training proceeds, a constant learning rate is a compromise across the whole run. Hence schedules.

• Warmup. Ramp η from near zero over the first few hundred to few thousand steps. At initialisation, gradients are large and unrepresentative, and an adaptive optimiser’s second-moment estimate is built from almost no samples, so its per-parameter scaling is unreliable exactly when the steps are biggest. The original transformer paper (Vaswani et al., 2017) used warmup followed by inverse-square-root decay, and some form of warmup has been standard for transformer training since.
• Cosine decay. Smoothly anneal from the peak to near zero over the planned number of steps. The current default for large model training. Note that it requires knowing the total step count in advance — stopping early leaves you at a high learning rate and a worse checkpoint than the schedule intended.
• Step decay. Multiply by 0.1 at fixed milestones. Old-fashioned, transparent, and still fine for fine-tuning.
• Reduce on plateau. Cut the rate when held-out loss stops improving. Requires no schedule planning and reacts to what actually happened, at the cost of being one evaluation behind.

Finding yours in 300 steps

You cannot compute k_max for a real network, but you can find the divergence point empirically. The range test — introduced by Smith (2017) in the cyclical learning rates work — increases η exponentially over a few hundred steps and records the loss:

lr, losses = 1e-7, []
for step, batch in enumerate(loader):
    for g in opt.param_groups:
        g["lr"] = lr
    loss = train_one_step(batch)
    losses.append((lr, loss))
    if loss > 4 * min(l for _, l in losses):
        break                      # diverging; stop
    lr *= 1.1                      # ~300 steps from 1e-7 to ~1e-1

Plot loss against η on a log axis. It will be flat, then descend, then turn sharply upward. Take the point where the descent is steepest, or roughly an order of magnitude below where it turns up. The test costs a few hundred steps and replaces a grid search, and it has to be redone whenever the batch size, the architecture or the optimiser changes — because all three change the curvature the bound depends on. Batch size in particular is not independent of this choice: see the scaling rules.

Two habits go with it. First, gradient clipping — rescaling the gradient whenever its norm exceeds a threshold — is insurance against the one bad batch whose gradient is large enough to push a stable run over the bound derived above. It does not substitute for a correct learning rate; it prevents a single outlier from destroying a run that was otherwise fine. Second, fine-tuning wants a learning rate one to two orders of magnitude below pretraining, and the reason is in the derivation: you are starting near a minimum rather than far from one, so the useful step size is the one that refines the solution rather than the one that travels to it. A fine-tune that forgets everything it knew is very often just a learning rate borrowed from a pretraining recipe.