Loss Functions: Choosing the Right One

A loss function has two jobs, and only the first is obvious. It turns “wrong” into a number — and it has to do so in a way whose derivative points somewhere useful, because the derivative is the only part the training loop ever sees.

What a loss is for

The optimiser never looks at your model’s output. It looks at ∂L/∂θ. Everything about a loss that matters is therefore a statement about its gradient: where it is large, where it is small, where it is zero, and which errors it treats as interchangeable. Two losses can rank predictions identically and train completely differently.

There is a second, quieter consequence. Minimising a loss makes the model converge to a specific statistic of the conditional distribution, and different losses converge to different ones. If you have not chosen deliberately, you have chosen anyway.

MSE finds the mean, MAE finds the median

This is provable in one line each, on the simplest case: predict a single constant c for a set of targets.

For squared error, L = Σ (c − yᵢ)², differentiate: dL/dc = 2Σ(c − yᵢ) = 0 gives c = (1/n)Σyᵢ — the mean. For absolute error, L = Σ |c − yᵢ|, the derivative of each term is +1 when c is above yᵢ and −1 below, so the sum is zero when equally many targets lie on each side — the median.

That is the whole outlier story, stated exactly. One target of 1,000,000 in a set otherwise near 10 drags the mean and does not move the median, so MSE chases it and MAE ignores it. The gradient shows the same thing from the other side: MSE’s gradient is proportional to the error, so a 100× larger error contributes a 100× larger push; MAE’s gradient is ±1 regardless of magnitude.

MAE’s constant gradient has a cost of its own: near the optimum, where errors are tiny, the step size does not shrink, so training can rattle around the minimum. Huber loss is the obvious join — quadratic within a distance δ of zero, linear outside it — which gives shrinking steps near the answer and bounded influence for outliers, with δ as the knob that says how large an error counts as an outlier.

The cross-entropy derivation

Classification with K classes. The model emits raw scores (logits) z₁ … z_K, softmax turns them into probabilities, and the loss is the negative log probability of the correct class y:

pⱼ = exp(zⱼ) / Σₖ exp(zₖ)
L  = −log p_y = −z_y + log Σₖ exp(zₖ)

The second line is worth pausing on: substituting softmax into the log collapses it, because log(a/b) = log a − log b. Now differentiate with respect to an arbitrary logit zⱼ. The first term contributes −1 if j is the correct class and 0 otherwise. The second term is the derivative of a log-sum-exp, which is:

∂/∂zⱼ log Σₖ exp(zₖ) = exp(zⱼ) / Σₖ exp(zₖ) = pⱼ

so   ∂L/∂zⱼ = pⱼ − 1[j = y]

In vector form: ∂L/∂z = p − y, where y is the one-hot target. The gradient at the output layer is literally prediction minus target. No softmax derivative survives, no division, nothing that can overflow.

Three consequences follow directly. The gradient is bounded in [−1, 1] per logit, so a confidently wrong prediction produces a large but not explosive update. A perfectly confident correct prediction produces exactly zero gradient, which is why label smoothing exists — it keeps a small pull alive so logits do not grow without limit. And frameworks fuse softmax and cross-entropy into one op precisely because the fused gradient is this clean; computing them separately is both slower and numerically worse.

This is also the loss that trains every language model you use. Next-token prediction is K-way classification with K equal to the vocabulary size, one classification per position. When a training log reports “loss 2.1”, that is this quantity in nats, and exp(2.1) ≈ 8.2 is the perplexity — the effective number of tokens the model was choosing between. The connection to the log probabilities an inference API can return is exact: a returned logprob for the chosen token is −L for that position.

Why accuracy cannot be a loss

The obvious question is why not optimise the thing you care about. The answer is calculus. Accuracy is a count of correct predictions, so it is a step function of the parameters: nudge a weight and either nothing changes or a prediction flips and accuracy jumps. Its derivative is zero almost everywhere and undefined at the jumps. There is nothing for gradient descent to follow.

So every training objective is a differentiable surrogate for the thing you actually want, and the gap between the two is a permanent feature of the field rather than a mistake. It is exactly why the metric you report is not the loss you train, and why a model can improve on one while getting worse on the other.

Reaching for the right one

One habit is worth more than the table: before choosing, write down the two errors your system can make and what each costs. If they cost differently, a symmetric loss is already the wrong choice, and weighting or threshold adjustment is not a hack but the correction.