Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 2 · Learning from data

Overfitting and regularisation

A model can be too simple or too flexible. Both do badly on new data, for different reasons.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

We fit curves of rising flexibility to the same 15 points and score each one on 200 test points, as in Lesson 9. The training error keeps falling. The test error does not.

Then we look at why, through bias and variance, and at one fix: a penalty on large coefficients, called ridge regression.

The points are made up. See "About the data" at the end.

You'll learn

  • what underfitting and overfitting look like, in the curve and in the scores
  • what bias and variance mean, and how they add up to the test error
  • how a ridge penalty, with strength λ, calms a model that is too flexible
  1. Too simple: a straight line

    A straight line cannot bend. It misses the hump and the dip, on the training points and the test points alike. This is underfitting.

    The model learns from the 15 cyan training points only. The 200 small violet squares are test points from the same source, held back as in Lesson 9. With 200 of them, the test MSE is a much steadier estimate than with 12.

    The model is a polynomial of degree p: ŷ = θ₀ + θ₁x + θ₂x² + … + θpxp, fitted by least squares on the training points. Degree 1 is a straight line.

    The chart on the right shows the training MSE (solid) and the test MSE (dashed) for every degree from 0 to 12. The upright dashed line marks the degree on show.

    The maths

    The data are y=sin⁡(πx)+εy = \sin(\pi x) + \varepsilon, with noise ε\varepsilon of mean 0 and SD 0.25. A polynomial model of degree pp is

    y^=θ0+θ1x+θ2x2+⋯+θpxp\hat{y} = \theta_0 + \theta_1 x + \theta_2 x^2 + \dots + \theta_p x^p

    The curve bends, but the prediction is a weighted sum of the features x,x2,…,xpx, x^2, \dots, x^p, so it is linear in θ\boldsymbol{\theta}. The training MSE is the same kind of bowl as in Lesson 8, step 6, and least squares finds its lowest point exactly.

  2. About right: degree 3

    A cubic can make one hump and one dip, the shape of the data. It follows the pattern without chasing each point.

    The test MSE is still above 0.0625. No curve can predict the noise in a new point, and this curve is not exactly the true one: sin(πx) is not a cubic.

    Degree 3 is a good fit, but not necessarily the best one. Step 4 compares every degree.

    The maths

    For a new point, the error has two parts: the noise ε\varepsilon, which no model can predict, and the gap between the fitted curve and sin⁡(πx)\sin(\pi x). The noise alone contributes 0.252=0.06250.25^2 = 0.0625 to the expected test MSE. Step 5 splits the gap into two parts as well.

  3. Too flexible: degree 12

    Degree 12 has 13 numbers to fit to 15 points. It bends to chase the noise, and between and beyond the points it swings far from the pattern. This is overfitting.

    With 13 numbers and 15 points, the curve cannot meet all the points exactly, but it bends towards many of them, noise and all.

    The maths

    A degree-pp polynomial has p+1p + 1 numbers. With nn points at different x values, degree n−1n - 1 can pass through every one of them exactly. Here n=15n = 15, so that would take degree 14. Degree 12 comes close to many points without meeting them all.

  4. Every degree at once

    Move the degree slider. The training error keeps falling as the degree rises. The test error falls at first, then rises.

    The chart shows both scores for every degree on a log scale: each labelled level is 100 times the one below it.

    The training MSE never goes up as the degree rises. A higher degree can always set its extra θ to 0 and copy the lower degree's curve, so least squares does at least as well. That holds because the loss here is the training MSE alone; step 6 adds something to it.

    We look at the test curve here only to see what happens. In practice you would choose the degree by validation, as in Lesson 9, and use the test set once at the end.

    The maths

    Every degree-pp curve is also a degree-(p+1)(p+1) curve with θp+1=0\theta_{p+1} = 0. So the lowest training MSE over degree p+1p + 1 is taken over a larger set of curves:

    min⁡deg⁡≤p+1MSEtrain  ≤  min⁡deg⁡≤pMSEtrain\min_{\deg \le p+1} \text{MSE}_{\text{train}} \;\le\; \min_{\deg \le p} \text{MSE}_{\text{train}}

    Nothing like this holds for the test MSE, which is measured on points the fit never saw.

  5. Bias and variance

    Too simple misses the pattern: that is bias. Too flexible learns the noise: that is variance. The expected test error adds the two, plus noise that no model can remove.

    Bias is how far the average fit sits from the true curve (dotted). With degree 1 the fits agree with one another, but all of them miss the bend.

    Variance is how much the fits disagree with one another. With degree 7 each fit follows its own noise, so they spread widely. Degree 12's fits spread much further still, too far to draw usefully. Degree 3 keeps both small.

    The maths

    Write f(x)=sin⁡(πx)f(x) = \sin(\pi x) for the true curve, f^\hat{f} for a fitted curve and σ2=0.252=0.0625\sigma^2 = 0.25^2 = 0.0625 for the variance of the noise. For squared loss at a fixed x0x_0, with a new y0=f(x0)+ε0y_0 = f(x_0) + \varepsilon_0,

    E[(y0−f^(x0))2]=(f(x0)−E[f^(x0)])2+Var⁡(f^(x0))+σ2\begin{aligned} &\mathbb{E}\bigl[(y_0 - \hat{f}(x_0))^2\bigr] \\ &\quad = \bigl(f(x_0) - \mathbb{E}[\hat{f}(x_0)]\bigr)^2 + \operatorname{Var}\bigl(\hat{f}(x_0)\bigr) + \sigma^2 \end{aligned}

    The expectation is over training sets and over the noise in y0y_0. The first term is the bias squared, the second the variance; σ2\sigma^2 is the irreducible noise, which even the true curve cannot avoid. Averaging over x0x_0 gives the expected test MSE.

    The stage estimates E[f^(x0)]\mathbb{E}[\hat{f}(x_0)] by the average fˉ(x0)\bar{f}(x_0) of the M=200M = 200 fits at x0x_0, and the variance by their population variance (dividing by MM). The average is itself noisy, so the squared gap of the average overstates bias² by about variance ÷ (number of fits):

    E[(f(x0)−fˉ(x0))2]=bias2+Var⁡(f^(x0))M\mathbb{E}\bigl[(f(x_0) - \bar{f}(x_0))^2\bigr] = \text{bias}^2 + \frac{\operatorname{Var}(\hat{f}(x_0))}{M}

    The stage subtracts that amount. When the squared gap is less than twice it, the bias² is reported as too small to measure with this many fits.

  6. Ridge: a penalty on big coefficients

    Keep degree 12, but add a penalty for large coefficients, the θs. Its strength is λ (lambda). The right λ calms the curve.

    Ridge regression chooses the θs to make the training MSE plus λ times (θ₁² + … + θ₁₂²) as small as possible. Wild swings need large coefficients, so the penalty pushes against them. θ₀ is not penalised, so the curve can still sit at any height.

    The slider runs from λ = 10⁻⁶ to 10 on a log scale: each notch multiplies λ by the same amount, about 1.26. The read-out also gives the largest of θ₁ to θ₁₂ in size.

    As λ grows, the training MSE rises: the penalty gives up some fit to the training points for a calmer curve. In practice λ is chosen like the degree, by validation.

    The maths

    With n=15n = 15 training points and residuals eie_i, ridge minimises

    J(θ)=1n∑i=1nei2+λ∑j=1pθj2J(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} e_i^2 + \lambda \sum_{j=1}^{p} \theta_j^2

    The sum starts at j=1j = 1, so the intercept θ0\theta_0 is free. Many books penalise the sum of squared errors instead of their mean; multiplying JJ by nn shows that their λ\lambda is nn times the one here, with the same fitted curve. The code solves that sum form with nλn\lambda. With the features in a matrix XX and D=diag⁡(0,1,…,1)D = \operatorname{diag}(0, 1, \dots, 1),

    θ^=(X⊤X+nλD)−1X⊤y\hat{\boldsymbol{\theta}} = \bigl(X^{\top}X + n\lambda D\bigr)^{-1} X^{\top}\mathbf{y}

    For any λ>0\lambda > 0 the bowl is steeper in every penalised direction and has exactly one lowest point.

    The penalty treats every coefficient alike, so it depends on the scale of the features. Here they are x,x2,…,x12x, x^2, \dots, x^{12} with xx in [−1,1][-1, 1], so all of them lie in [−1,1][-1, 1]. Features on very different scales are usually standardised first (Lesson 8).

Quick check

Which degree has the lowest training MSE, and why is that misleading?

Degree 3 fits the pattern well, but every higher degree can do at least as well on the training points, so its training MSE is not the lowest.

A straight line has one of the highest training MSEs here; only degree 0 is higher. Simple is not the same as a close fit.

Right. With no penalty, the training MSE never rises with the degree, so the most flexible curve scores best on the points it learned from. That is exactly the optimism Lesson 9 warned about.

They do not agree: the training MSE keeps falling while the test MSE turns back up.

What does a very large λ do to the degree-12 ridge fit?

Right. The penalty outweighs the fit, so the slopes shrink. The intercept is not penalised, so the curve flattens towards the mean of y.

θ₀ is not penalised, so the curve keeps its height. It flattens towards the mean of the training y values, not towards 0.

The training MSE rises as λ grows. The penalty trades fit to the training points for smaller coefficients.

All 13 numbers stay in the model. Ridge shrinks them towards 0 but does not, in general, make any of them exactly 0.

Lesson complete

  • Too simple a model misses the pattern (underfitting). Too flexible a model learns the noise (overfitting). Both do badly on new data.
  • With no penalty, the training error never rises with the degree. The test error typically falls and then rises.
  • The expected test error at a point is bias squared plus variance plus the noise variance σ².
  • Ridge adds λ times the sum of the squared coefficients, leaving the intercept out. A well-chosen λ tames a flexible model.

Common misconceptions

  • "A more flexible model is always better." It is at least as good on the training points (with no penalty), which is not what matters.
  • "Overfitting shows up as a high training error." The training error is low. The sign is the gap between it and the test error.
  • "More regularisation is always safer." Too large a λ flattens the curve and underfits.
Next lesson: k-nearest neighbours →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.2.2, "The Bias-Variance Trade-Off", and Section 6.2.1, "Ridge Regression". Its ridge penalty is added to the sum of squared errors, so its λ is n times the one here.
  2. Goodfellow, Bengio & Courville, Deep Learning, Section 5.2, "Capacity, Overfitting and Underfitting", which compares linear, quadratic and degree-9 polynomial fits. Its Section 5.2.2, "Regularization", adds a weight decay penalty to the training MSE, in the same form as the ridge objective here, apart from leaving out the intercept.

About the data. The points are made up with the same recipe as Lesson 9. For each point in turn, x is drawn uniformly between −1 and 1, then y = sin(πx) plus 0.25 times a draw from a normal distribution with mean 0 and SD 1, using mulberry32. The 15 training points use seed 3, so they are the first 15 of Lesson 9's 40 points; the 200 test points use seed 4. Step 5's extra training sets of 15 points use seeds 101 to 300; the spread quoted for degree 7 also uses five more blocks of 200 seeds, starting at 301, 501, 701, 901 and 1101. The points miss the wave only because of that noise; real data also has measurement error and things the model leaves out, and usually no known true curve. Fits use least squares, with the ridge penalty in step 6. Displayed values are rounded; the calculations are not.

If this helped, you can .