ML Explorer · Level 2 · Learning from data
Overfitting and regularisation
A model can be too simple or too flexible. Both do badly on new data, for different reasons.
Overview
We fit curves of rising flexibility to the same 15 points and score each one on 200 test points, as in Lesson 9. The training error keeps falling. The test error does not.
Then we look at why, through bias and variance, and at one fix: a penalty on large coefficients, called ridge regression.
The points are made up. See "About the data" at the end.
You'll learn
- what underfitting and overfitting look like, in the curve and in the scores
- what bias and variance mean, and how they add up to the test error
- how a ridge penalty, with strength λ, calms a model that is too flexible
-
Too simple: a straight line
A straight line cannot bend. It misses the hump and the dip, on the training points and the test points alike. This is underfitting.
The model learns from the 15 cyan training points only. The 200 small violet squares are test points from the same source, held back as in Lesson 9. With 200 of them, the test MSE is a much steadier estimate than with 12.
The model is a polynomial of degree p: ŷ = θ₀ + θ₁x + θ₂x² + … + θpxp, fitted by least squares on the training points. Degree 1 is a straight line.
Training MSE , test MSE . Both are well above 0.0625, the variance of the noise (its SD, 0.25, squared). The line is too simple for the pattern.
The chart on the right shows the training MSE (solid) and the test MSE (dashed) for every degree from 0 to 12. The upright dashed line marks the degree on show.
The maths
The data are , with noise of mean 0 and SD 0.25. A polynomial model of degree is
The curve bends, but the prediction is a weighted sum of the features , so it is linear in . The training MSE is the same kind of bowl as in Lesson 8, step 6, and least squares finds its lowest point exactly.
-
About right: degree 3
A cubic can make one hump and one dip, the shape of the data. It follows the pattern without chasing each point.
Training MSE , test MSE . Both are far lower than the line's, and close to each other.
The test MSE is still above 0.0625. No curve can predict the noise in a new point, and this curve is not exactly the true one: sin(πx) is not a cubic.
Degree 3 is a good fit, but not necessarily the best one. Step 4 compares every degree.
The maths
For a new point, the error has two parts: the noise , which no model can predict, and the gap between the fitted curve and . The noise alone contributes to the expected test MSE. Step 5 splits the gap into two parts as well.
-
Too flexible: degree 12
Degree 12 has 13 numbers to fit to 15 points. It bends to chase the noise, and between and beyond the points it swings far from the pattern. This is overfitting.
The training MSE falls to , the lowest of any degree here. The test MSE jumps to .
With 13 numbers and 15 points, the curve cannot meet all the points exactly, but it bends towards many of them, noise and all.
The test points come from the whole range −1 to 1, the same source as the training points, but by chance none of the 15 training points fell past x = . There, and in the empty gap around x = , nothing holds the curve down, and that is where most of its error comes from. Even on the test points inside the training range, its test MSE is , against for degree 3.
The maths
A degree- polynomial has numbers. With points at different x values, degree can pass through every one of them exactly. Here , so that would take degree 14. Degree 12 comes close to many points without meeting them all.
-
Every degree at once
Move the degree slider. The training error keeps falling as the degree rises. The test error falls at first, then rises.
The chart shows both scores for every degree on a log scale: each labelled level is 100 times the one below it.
The training MSE never goes up as the degree rises. A higher degree can always set its extra θ to 0 and copy the lower degree's curve, so least squares does at least as well. That holds because the loss here is the training MSE alone; step 6 adds something to it.
The test MSE is typically U-shaped: high when the model is too simple, lowest in between, high again when it is too flexible. Here it is lowest at degree (test MSE ), and it is not a smooth U: past that it jumps up and down as it climbs. With only 15 training points, each degree's curve depends a lot on their noise.
We look at the test curve here only to see what happens. In practice you would choose the degree by validation, as in Lesson 9, and use the test set once at the end.
The maths
Every degree- curve is also a degree- curve with . So the lowest training MSE over degree is taken over a larger set of curves:
Nothing like this holds for the test MSE, which is measured on points the fit never saw.
-
Bias and variance
Too simple misses the pattern: that is bias. Too flexible learns the noise: that is variance. The expected test error adds the two, plus noise that no model can remove.
Imagine collecting a fresh set of 15 points many times, and fitting the same degree each time. The stage does this times. The thin lines are of the fits; the thick line is the average of all of them.
Bias is how far the average fit sits from the true curve (dotted). With degree 1 the fits agree with one another, but all of them miss the bend.
Variance is how much the fits disagree with one another. With degree 7 each fit follows its own noise, so they spread widely. Degree 12's fits spread much further still, too far to draw usefully. Degree 3 keeps both small.
At x₀ = , the dashed line: degree 1 has bias² and variance . Degree 3 has variance , and its bias² is . Degree 7 has variance , and its bias² is . These are estimates from the training sets.
A few wild fits dominate degree 7's variance. In six blocks of training sets it ranged from about to , but was always far above degree 3's.
The maths
Write for the true curve, for a fitted curve and for the variance of the noise. For squared loss at a fixed , with a new ,
The expectation is over training sets and over the noise in . The first term is the bias squared, the second the variance; is the irreducible noise, which even the true curve cannot avoid. Averaging over gives the expected test MSE.
The stage estimates by the average of the fits at , and the variance by their population variance (dividing by ). The average is itself noisy, so the squared gap of the average overstates bias² by about variance ÷ (number of fits):
The stage subtracts that amount. When the squared gap is less than twice it, the bias² is reported as too small to measure with this many fits.
-
Ridge: a penalty on big coefficients
Keep degree 12, but add a penalty for large coefficients, the θs. Its strength is λ (lambda). The right λ calms the curve.
Ridge regression chooses the θs to make the training MSE plus λ times (θ₁² + … + θ₁₂²) as small as possible. Wild swings need large coefficients, so the penalty pushes against them. θ₀ is not penalised, so the curve can still sit at any height.
The slider runs from λ = 10⁻⁶ to 10 on a log scale: each notch multiplies λ by the same amount, about 1.26. The read-out also gives the largest of θ₁ to θ₁₂ in size.
At λ = the test MSE is , already far below the with no penalty. Unpenalised, degree 12's largest coefficient is about in size, so even a tiny penalty bites. The lowest test MSE on the slider is , at λ = . At λ = 10 it is back up to : the curve has flattened towards a horizontal line and underfits.
As λ grows, the training MSE rises: the penalty gives up some fit to the training points for a calmer curve. In practice λ is chosen like the degree, by validation.
The maths
With training points and residuals , ridge minimises
The sum starts at , so the intercept is free. Many books penalise the sum of squared errors instead of their mean; multiplying by shows that their is times the one here, with the same fitted curve. The code solves that sum form with . With the features in a matrix and ,
For any the bowl is steeper in every penalised direction and has exactly one lowest point.
The penalty treats every coefficient alike, so it depends on the scale of the features. Here they are with in , so all of them lie in . Features on very different scales are usually standardised first (Lesson 8).
Quick check
Lesson complete
- Too simple a model misses the pattern (underfitting). Too flexible a model learns the noise (overfitting). Both do badly on new data.
- With no penalty, the training error never rises with the degree. The test error typically falls and then rises.
- The expected test error at a point is bias squared plus variance plus the noise variance σ².
- Ridge adds λ times the sum of the squared coefficients, leaving the intercept out. A well-chosen λ tames a flexible model.
Common misconceptions
- "A more flexible model is always better." It is at least as good on the training points (with no penalty), which is not what matters.
- "Overfitting shows up as a high training error." The training error is low. The sign is the gap between it and the test error.
- "More regularisation is always safer." Too large a λ flattens the curve and underfits.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.2.2, "The Bias-Variance Trade-Off", and Section 6.2.1, "Ridge Regression". Its ridge penalty is added to the sum of squared errors, so its λ is n times the one here.
- Goodfellow, Bengio & Courville, Deep Learning, Section 5.2, "Capacity, Overfitting and Underfitting", which compares linear, quadratic and degree-9 polynomial fits. Its Section 5.2.2, "Regularization", adds a weight decay penalty to the training MSE, in the same form as the ridge objective here, apart from leaving out the intercept.
About the data. The points are made up with the same recipe as Lesson 9. For each point in turn, x is drawn uniformly between −1 and 1, then y = sin(πx) plus 0.25 times a draw from a normal distribution with mean 0 and SD 1, using mulberry32. The 15 training points use seed 3, so they are the first 15 of Lesson 9's 40 points; the 200 test points use seed 4. Step 5's extra training sets of 15 points use seeds 101 to 300; the spread quoted for degree 7 also uses five more blocks of 200 seeds, starting at 301, 501, 701, 901 and 1101. The points miss the wave only because of that noise; real data also has measurement error and things the model leaves out, and usually no known true curve. Fits use least squares, with the ridge penalty in step 6. Displayed values are rounded; the calculations are not.
If this helped, you can .