ML Explorer · Level 2 · Learning from data
Train, validate, test
A model is only useful if it works on data it has not seen. Here is how to measure that honestly.
Overview
So far every lesson has scored a model on the same points it learned from. That score flatters the model. This lesson holds some points back, scores the model on them, and shows how to choose a setting without fooling yourself.
The points are made up. See "About the data" at the end.
You'll learn
- why the error on the training data is usually too optimistic
- what a test set is, and why one test score is only an estimate
- how a validation set or k-fold cross-validation chooses settings, leaving the test set for the end
-
A score on its own data flatters a model
A rule can fit the points it learned from perfectly and still be a poor guide to new points. Its score on those points is optimistic.
Each of the 40 points is a smooth wave, y = sin(πx), plus some random noise. We made them, so we know the true curve: the dotted line.
The grey line is a rule that joins the dots. Between two neighbouring points it predicts along the straight line that joins them. On these 40 points every prediction is exact, so its mean squared error (MSE, Lesson 4) is 0.
The true curve scores worse on the same points: MSE . It misses each point by exactly that point's noise.
Yet for a new point the true curve is the better guess. The new point brings its own fresh noise, and joining the dots has copied the old noise. A score on the points a model learned from cannot show you this.
The maths
For points , , and predictions ,
The data are , where the noise has mean 0 and SD 0.25. For the true curve, , so its MSE is the mean of the squared noise. Its expected value is , though any 40 draws give a value a little above or below it.
-
Hold some points back
Before fitting, set some points aside. The model learns from the training set. The test set plays the part of new data.
Here 30% of the 40 points, 12 of them, are picked at random to be the test set: the violet squares. The other 28, the cyan dots, are the training set.
The model never sees the test points while it learns. Afterwards, its error on them estimates its error on new points from the same source.
The split is random so that the test points are like new data. Points picked by hand, say all those with large x, would test something else.
The maths
Number the points 1 to 40 and shuffle the numbers at random, every order equally likely. The first 12 form the test set ; the rest form the training set . The two scores are
where and count the points in each set.
-
Fit on training, score on both
Fit a curve to the 28 training points only. Then score it twice: on the training points and on the test points.
The curve is a polynomial of degree 3: ŷ = θ₀ + θ₁x + θ₂x² + θ₃x³. Its four numbers θ₀ to θ₃ are chosen by least squares on the training points, the same idea as the line in Lesson 6.
On this split the training MSE is and the test MSE is .
On this split the training MSE is and the test MSE is : this time the training score is the higher one. With only 12 test points, a lucky set of easy points can do that.
The dashed sticks show how far the curve misses each test point. The training score is usually the lower one. The curve was chosen to fit those very points, noise and all, so their score is optimistic. The test points had no say in the curve, so their score is the honest one.
The maths
The curve's numbers minimise the training MSE:
The curve bends, but the prediction is still a weighted sum of the features , so it is linear in . The loss is the same kind of bowl as in Lesson 8, step 6, and least squares solves it.
-
One split is one noisy estimate
A different random split gives a different test score. One test MSE is an estimate of the error on new data, not its exact value.
Press New split. The 40 points stay the same; only the choice of the 12 test points changes, and the curve is refitted.
The strip shows both scores for the first splits, with the current split ringed. The test MSE runs from to . In of them the training MSE is the lower one; in the rest it is higher.
Over the first splits, the training MSE was lower in , and on average it was against for the test MSE. That is what "usually optimistic" means.
The split now showing is one of those: its training MSE, , is above its test MSE, .
With only 12 test points, the estimate swings a lot. A larger test set swings less.
The maths
For one fitted curve and independent test points from the same source, the test MSE is a mean of independent squared errors. Its SD is the SD of one squared error divided by , so four times as many test points halve the swing.
Across splits the curve changes as well, which adds to the swing you see here.
-
Choose with validation, test once
To choose a setting such as the degree, compare the options on validation data taken from the training set. Keep the test set for one score at the very end.
Suppose you tried degrees 1, 2, 3 and so on, and kept whichever scored best on the test set. The test set would then have helped pick the model, and its score would tend to be optimistic for the same reason as a training score.
A validation set is a part of the training data held back to compare settings. k-fold cross-validation does this more thoroughly. Split the training points into k parts; here k = 5. Each part takes one turn as the validation set, while the model is fitted on the other four. Then average the k validation scores.
The bars show that average for degrees 1 to 6. Degree scores lowest. It is refitted on all 28 training points and only then scored on the test set, once: test MSE .
Try New split in step 4 and come back: the chosen degree can change with the split. That is the same noise again.
The maths
Split into parts ; here they hold 5 or 6 points each. For part , fit on without and compute the MSE on , written . The cross-validation score is
Every training point is in the validation part exactly once. This plain average treats the parts as equal; weighting each by its size is another common choice.
Quick check
Lesson complete
- A model's error on the points it learned from is usually lower than its error on new points. It is optimistic.
- A test set, held back at random, gives an honest estimate of the error on new data. With few test points, that estimate is noisy.
- Choose settings with a validation set or k-fold cross-validation, inside the training data. Score on the test set once, at the end.
Common misconceptions
- "A low training error means a good model." Joining the dots scores 0 on its own points and copies their noise.
- "The test MSE is the model's true error." It is an estimate from a handful of points, and another split gives another number.
- "Cross-validation uses the test set." It works inside the training data. The test set stays untouched until the end.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.2.1, "Measuring the Quality of Fit", on training MSE against test MSE, and Section 5.1, "Cross-Validation", which covers the validation set approach and k-fold cross-validation.
- Goodfellow, Bengio & Courville, Deep Learning, Section 5.3, "Hyperparameters and Validation Sets", including Section 5.3.1, "Cross-Validation".
About the data. The 40 points are made up, from a seeded random number generator, mulberry32 with seed 3. For each point in turn, x is drawn uniformly between −1 and 1, then y = sin(πx) plus 0.25 times a draw from a normal distribution with mean 0 and SD 1. The points miss the wave only because of that noise; real data also has measurement error and things the model leaves out, and usually no known true curve. The first split shuffles the point numbers with a separate mulberry32 generator, also seeded 3; each New split adds 1 to that seed. The 5 cross-validation parts are consecutive runs of the shuffled training points. Displayed values are rounded; the calculations are not.
If this helped, you can .