Overfitting Demonstrator
Drag polynomial degree and watch training error fall to zero while held-out error turns upward.
Training error is 0.1441. The noise you added has a standard deviation of 0.150, and no model can score below that on held-out data — the best degree here is 3, at 0.1471.
- Degree
- 3
- Free parameters
- 4
- Training points
- 15
- Training RMSE
- 0.1441
- Held-out RMSE
- 0.1471
- Held-out ÷ training
- 1.02×
- Noise floor (the σ you set)
- 0.1500
- Degree with the lowest held-out error
- 3
Overfitting is not the model being wrong. At degree 11 with fifteen points the model is perfect on everything it has seen — the blue training curve is on the floor — and useless on anything it has not. The two lines in the lower chart separating is the whole definition, and it is worth seeing that the separation is not gradual: held-out error is flat and boring for several degrees, then turns upward sharply once the model has enough freedom to chase individual noise samples.
The dashed noise floor is the second thing to take away. No model, of any size, trained on any amount of this data, can score below it on held-out points, because the noise is not a function of x and there is nothing to learn about it. A held-out error near the floor means you are done; a training error below the floor means you have started memorising. That gap — not the absolute number — is the diagnostic.
Raise the training points to 50 and drag the degree back up. The turn arrives much later, because overfitting is a relationship between model capacity and data volume rather than a property of the model. That is the same lever behind every practical fix: more data, fewer effective parameters, or regularisation, which shrinks capacity without removing it. What this cannot show is the modern case where enormously overparameterised models generalise anyway — a real and unresolved exception to the U-curve, not a repeal of it.