Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 2 · Learning from data

Train, validate, test

A model is only useful if it works on data it has not seen. Here is how to measure that honestly.

About 10 minutes · No maths needed; the maths is there if you want it

Overview

So far every lesson has scored a model on the same points it learned from. That score flatters the model. This lesson holds some points back, scores the model on them, and shows how to choose a setting without fooling yourself.

The points are made up. See "About the data" at the end.

You'll learn

  • why the error on the training data is usually too optimistic
  • what a test set is, and why one test score is only an estimate
  • how a validation set or k-fold cross-validation chooses settings, leaving the test set for the end
  1. A score on its own data flatters a model

    A rule can fit the points it learned from perfectly and still be a poor guide to new points. Its score on those points is optimistic.

    Each of the 40 points is a smooth wave, y = sin(πx), plus some random noise. We made them, so we know the true curve: the dotted line.

    The grey line is a rule that joins the dots. Between two neighbouring points it predicts along the straight line that joins them. On these 40 points every prediction is exact, so its mean squared error (MSE, Lesson 4) is 0.

    Yet for a new point the true curve is the better guess. The new point brings its own fresh noise, and joining the dots has copied the old noise. A score on the points a model learned from cannot show you this.

    The maths

    For points (xi,yi)(x_i, y_i), i=1,…,ni = 1, \dots, n, and predictions y^i\hat{y}_i,

    MSE=1n∑i=1nei2,ei=yi−y^i\text{MSE} = \frac{1}{n}\sum_{i=1}^{n} e_i^2, \qquad e_i = y_i - \hat{y}_i

    The data are yi=sin⁡(πxi)+εiy_i = \sin(\pi x_i) + \varepsilon_i, where the noise εi\varepsilon_i has mean 0 and SD 0.25. For the true curve, ei=εie_i = \varepsilon_i, so its MSE is the mean of the squared noise. Its expected value is 0.252=0.06250.25^2 = 0.0625, though any 40 draws give a value a little above or below it.

  2. Hold some points back

    Before fitting, set some points aside. The model learns from the training set. The test set plays the part of new data.

    Here 30% of the 40 points, 12 of them, are picked at random to be the test set: the violet squares. The other 28, the cyan dots, are the training set.

    The model never sees the test points while it learns. Afterwards, its error on them estimates its error on new points from the same source.

    The split is random so that the test points are like new data. Points picked by hand, say all those with large x, would test something else.

    The maths

    Number the points 1 to 40 and shuffle the numbers at random, every order equally likely. The first 12 form the test set TT; the rest form the training set RR. The two scores are

    MSEtrain=1∣R∣∑i∈Rei2MSEtest=1∣T∣∑i∈Tei2\begin{aligned} \text{MSE}_{\text{train}} &= \frac{1}{|R|}\sum_{i \in R} e_i^2 \\ \text{MSE}_{\text{test}} &= \frac{1}{|T|}\sum_{i \in T} e_i^2 \end{aligned}

    where ∣R∣=28|R| = 28 and ∣T∣=12|T| = 12 count the points in each set.

  3. Fit on training, score on both

    Fit a curve to the 28 training points only. Then score it twice: on the training points and on the test points.

    The curve is a polynomial of degree 3: ŷ = θ₀ + θ₁x + θ₂x² + θ₃x³. Its four numbers θ₀ to θ₃ are chosen by least squares on the training points, the same idea as the line in Lesson 6.

    The dashed sticks show how far the curve misses each test point. The training score is usually the lower one. The curve was chosen to fit those very points, noise and all, so their score is optimistic. The test points had no say in the curve, so their score is the honest one.

    The maths

    The curve's numbers minimise the training MSE:

    θ^=arg⁡min⁡θ1∣R∣∑i∈R(yi−θ0−θ1xi−θ2xi2−θ3xi3)2\hat{\boldsymbol{\theta}} = \arg\min_{\boldsymbol{\theta}} \frac{1}{|R|}\sum_{i \in R} \bigl(y_i - \theta_0 - \theta_1 x_i - \theta_2 x_i^2 - \theta_3 x_i^3\bigr)^2

    The curve bends, but the prediction is still a weighted sum of the features x,x2,x3x, x^2, x^3, so it is linear in θ\boldsymbol{\theta}. The loss is the same kind of bowl as in Lesson 8, step 6, and least squares solves it.

  4. One split is one noisy estimate

    A different random split gives a different test score. One test MSE is an estimate of the error on new data, not its exact value.

    Press New split. The 40 points stay the same; only the choice of the 12 test points changes, and the curve is refitted.

    With only 12 test points, the estimate swings a lot. A larger test set swings less.

    The maths

    For one fitted curve and independent test points from the same source, the test MSE is a mean of ∣T∣|T| independent squared errors. Its SD is the SD of one squared error divided by ∣T∣\sqrt{|T|}, so four times as many test points halve the swing.

    Across splits the curve changes as well, which adds to the swing you see here.

  5. Choose with validation, test once

    To choose a setting such as the degree, compare the options on validation data taken from the training set. Keep the test set for one score at the very end.

    Suppose you tried degrees 1, 2, 3 and so on, and kept whichever scored best on the test set. The test set would then have helped pick the model, and its score would tend to be optimistic for the same reason as a training score.

    A validation set is a part of the training data held back to compare settings. k-fold cross-validation does this more thoroughly. Split the training points into k parts; here k = 5. Each part takes one turn as the validation set, while the model is fitted on the other four. Then average the k validation scores.

    Try New split in step 4 and come back: the chosen degree can change with the split. That is the same noise again.

    The maths

    Split RR into parts R1,…,RkR_1, \dots, R_k; here they hold 5 or 6 points each. For part jj, fit on RR without RjR_j and compute the MSE on RjR_j, written MSEj\text{MSE}_j. The cross-validation score is

    CVk=1k∑j=1kMSEj\text{CV}_k = \frac{1}{k}\sum_{j=1}^{k} \text{MSE}_j

    Every training point is in the validation part exactly once. This plain average treats the parts as equal; weighting each by its size is another common choice.

Quick check

Why not choose the degree by trying each one on the test set?

Speed is not the problem. Scoring a fitted curve on 12 points is quick.

Right. Once the test points have been used to choose, they are no longer new data. Use a validation set or cross-validation to choose, and the test set once at the end.

Nothing is fitted on the test set either way. The trouble is using its score to choose.

It may pick a fine model. What breaks is the score: it no longer tells you honestly how the model does on new data.

In 5-fold cross-validation, how often is each training point in the validation part?

Every part takes a turn as the validation set, so every training point is validated on.

Right. The five parts do not overlap and each takes one turn. In the other four rounds the point is used for fitting.

Four is how often each point is used for fitting: in every round except its own part's turn.

There are five rounds, but in four of them the point is used for fitting, not for validation.

Lesson complete

  • A model's error on the points it learned from is usually lower than its error on new points. It is optimistic.
  • A test set, held back at random, gives an honest estimate of the error on new data. With few test points, that estimate is noisy.
  • Choose settings with a validation set or k-fold cross-validation, inside the training data. Score on the test set once, at the end.

Common misconceptions

  • "A low training error means a good model." Joining the dots scores 0 on its own points and copies their noise.
  • "The test MSE is the model's true error." It is an estimate from a handful of points, and another split gives another number.
  • "Cross-validation uses the test set." It works inside the training data. The test set stays untouched until the end.
Next lesson: Overfitting and regularisation →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.2.1, "Measuring the Quality of Fit", on training MSE against test MSE, and Section 5.1, "Cross-Validation", which covers the validation set approach and k-fold cross-validation.
  2. Goodfellow, Bengio & Courville, Deep Learning, Section 5.3, "Hyperparameters and Validation Sets", including Section 5.3.1, "Cross-Validation".

About the data. The 40 points are made up, from a seeded random number generator, mulberry32 with seed 3. For each point in turn, x is drawn uniformly between −1 and 1, then y = sin(πx) plus 0.25 times a draw from a normal distribution with mean 0 and SD 1. The points miss the wave only because of that noise; real data also has measurement error and things the model leaves out, and usually no known true curve. The first split shuffles the point numbers with a separate mulberry32 generator, also seeded 3; each New split adds 1 to that seed. The 5 cross-validation parts are consecutive runs of the shuffled training points. Displayed values are rounded; the calculations are not.

If this helped, you can .