ML Explorer · Level 1 · Data & intuition
Measuring error
Every model misses. A loss sums up all the misses in one number.
Overview
A model's predictions miss. To say how good a model is, or to train one, we need a single number that sums up how badly it misses. That number is called a loss. Smaller is better.
This lesson takes 11 points and one fixed line, and builds two losses from the line's misses: the mean absolute error (MAE) and the mean squared error (MSE). Lesson 1 promised to explain why we square. This is where it does.
The points are made up. See "About the data" at the end.
You'll learn
- what a residual is, and why residuals can't simply be added up
- how MAE and MSE are built, and how each one treats a single big miss
- why squaring is the usual choice, and which single guess each loss prefers
-
Residuals: how far each point misses
A residual is a miss: the real y minus the line's prediction.
Each vertical stick shows one residual, from a point to the line.
The line is the least-squares line for these 11 points, the "best fit" from Lesson 1. It stays fixed for the whole lesson. We are measuring its misses, not moving it.
For each point, the line gives a prediction, written ŷ and said "y-hat". The residual is y − ŷ. It is positive when the point sits above the line and negative when it sits below.
The ringed point sits above the line. The box gives its y, its ŷ and its residual, e.
The stage rounds to two decimal places.
The maths
The points are for , with . The fixed line is , so point has prediction and residual
The sign matters: means the line predicted too low, and means it predicted too high.
For the ringed point, = .
-
Why not just add them up?
Adding the residuals sounds like a fair total. But misses above the line are positive and misses below are negative, so they cancel.
The chart on the right lays the sticks end to end. The ones above the line go up (solid). Then the ones below come back down (dashed).
The residuals above the line add up to . The below add up to .
The walk ends where it started: the residuals add up to zero, yet every point misses.
That is no accident. For a least-squares line like this one, the residuals always add up to zero. So the plain sum tells you nothing about how well the line fits.
We need a total in which every miss counts against the line, whatever its sign. There are two common fixes: use the size of each miss, or its square.
The maths
For any line , with and the means of the and the ,
This is zero for every line that passes through the point , however badly it is tilted. Lesson 1's formula puts the least-squares line through that point, so its residuals add up to zero.
-
MAE: the average size of a miss
Drop the sign and keep the size. Average the sizes of the misses, and you have the mean absolute error, or MAE.
The size of a number, ignoring its sign, is its absolute value, written |e|. So the absolute value of −2 is 2, and so is the absolute value of 2.
The chart on the right stands every stick upright, in the order the points sit from left to right. The dashed bars hung below the line.
The dashed line across the chart is the average height of the bars. That is the MAE.
The sizes add up to , and / = . MAE is in the same units as y: on average, the line misses by about .
Every miss now counts against the line, and a miss twice as big counts twice as much.
The maths
The absolute value is when , and when . Then
-
MSE: square each miss
Or square each miss and average the squares. That is the mean squared error, or MSE: the "average squared miss" from Lesson 1.
Squaring a miss gives the area of a square with that side. Each stick now carries its square.
On the right, the bars change from each |e| to each e². The big misses grow a lot. Misses smaller than 1 shrink: 0.5 squared is 0.25.
Why square? Two reasons show here. A square is never negative, so misses cannot cancel. And big misses count for far more than small ones: a miss twice as big gets a square four times as big. Step 6 adds three more reasons.
The squares add up to , and / = .
MSE is in squared units of y, which is hard to picture. Step 6 shows how to get back to the units of y.
The maths
The makes it an average, so MSE does not grow just because there are more points.
Which misses count as "smaller than 1" depends on the units of y: measure in centimetres instead of metres and every miss is 100 times bigger. So that detail is not deep. What does not depend on units is that squaring widens the gap between small and big misses: if one miss is times another, its square is times the other's square.
-
One big miss
Drag the ringed point away from the line. Its square grows much faster than its stick.
The line stays where it is. Only this point moves, so only its residual changes.
When its miss becomes k times as big, its part of the MAE becomes k times as big too: it grows in proportion. Its part of the MSE becomes k² times as big: 3 times the miss, 9 times the square.
The chart on the right shows both. The dashed line is k and the solid curve is k².
Its residual starts at . The button makes it : three times the miss, so nine times the square.
The MSE as a whole grows by less than nine times, because the other ten squares stay the same. The box shows both losses, before and after.
This is what "MSE punishes big misses" means: one far-off point can dominate it. Whether that is what you want depends on the problem.
Drag the point, or Tab to it and use the arrow keys. "Put it back" returns it, and it also goes back when you leave this step.
The maths
Split off the ringed point, number :
The line is fixed, so dragging point changes only . Replace by , with . Its part of the MAE goes from to . Its part of the MSE goes from to .
The MAE rises in a straight line as grows. The MSE rises along a parabola. The other terms do not change, so the whole MSE grows by less than times when .
Dragging the point across the line flips the sign of . The stage counts the size, , and stops at .
-
The best constant guess
Suppose you had to guess one number, c, for every point, ignoring x. Which c is best? It depends on the loss: the mean of y for MSE, the median of y for MAE.
The line is now flat at height c, and each stick is a point's miss from c. Move the slider and watch the two curves on the right. Each curve gives the loss for every possible c, and the dots mark the current one.
The MSE is lowest when c is the mean of y, . The MAE is lowest when c is the median of y, . These are the mean and median from Lesson 3.
There are 11 points, an odd number, so the median is a single middle value and the lowest MAE is at that one c. With an even number of points, every c between the two middle values would give the same lowest MAE.
The chart shows the RMSE, the square root of the MSE, so that both curves are in the units of y. The MSE itself is in squared units. The RMSE is lowest exactly where the MSE is.
Look at the shapes. The MSE curve is smooth. The MAE curve has a corner at each value of y.
So why square? It stops misses cancelling, and it punishes big misses more. It is smooth, so it has a slope everywhere, which makes it easy to follow downhill with gradient descent (Lesson 7). It leads to a formula for the best line (Lesson 6). And its best constant guess is the familiar mean.
MAE has its own strength: one far-out point sways it less, just as one far-out value moves the median less than the mean.
The maths
For a constant guess, for every . Write for the mean of the . The MSE splits into two parts:
The first part does not depend on , and the second is zero only at . So the mean is the one best constant under MSE, and the lowest MSE is the population variance of y (divide by ). Lesson 3 used the same fact when it explained .
Here the lowest MSE is , at = . At the median, the MSE is .
For MAE, take between two neighbouring values of y, with values below it and above. Raising a little brings it closer to the values above and further from the below, so the slope of the MAE is . It is negative while fewer than half the values lie below and positive once more than half do. With odd, the MAE is therefore lowest at the middle value, the median, and nowhere else. With even, the slope is 0 between the two middle values, and every there ties. The slope jumps at each value of y, which is why the curve has corners.
Here the lowest MAE is , at = . At the mean, the MAE is .
Since and the square root only ever increases, the RMSE and the MSE are lowest at the same .
One more reason for squaring comes from probability. Suppose , where the are independent draws from a normal distribution with mean 0 and a fixed SD. The likelihood is the probability density of the observed , seen as a function of and . Its logarithm is a constant minus a positive multiple of , so the line with the highest likelihood is the least-squares line. These points were made exactly that way (see About the data).
Quick check
Lesson complete
- A residual is a miss, y − ŷ. Positive and negative residuals cancel, so their plain sum hides how badly a line fits.
- MAE averages the sizes of the misses. MSE averages their squares, so one big miss counts for much more.
- For a single constant guess, MSE is lowest at the mean and MAE at the median.
Common misconceptions
- "Residuals that add up to zero mean a perfect fit." The least-squares line always makes them add up to zero, however big the misses are.
- "MSE is in the same units as y." It is in squared units. Its square root, the RMSE, is in the units of y.
- "MAE and MSE always agree on the best answer." For a single constant guess, MSE picks the mean and MAE picks the median, and these often differ.
- "Squaring is the only correct way to measure error." It is a choice, with good reasons behind it. When a few far-off points should not dominate, MAE can suit better.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.2.1, "Measuring the Quality of Fit", which introduces the MSE as the most common measure of fit in regression.
- Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 9.2, "Parameter Estimation", which shows how maximum likelihood with normal noise leads to the squared-error loss, and notes that the RMSE has the same units as y.
- Goodfellow, Bengio & Courville, Deep Learning, Section 5.5.1, "Conditional Log-Likelihood and Mean Squared Error", which links the mean squared error to maximum likelihood for linear regression.
About the data. The 11 points are made up. They come from a seeded random number generator, mulberry32 with seed 5. For each point in turn, x is drawn uniformly between 0 and 10, and then y = 2 + 0.8x plus a random draw from a normal distribution with mean 0 and SD 1. So the points are the same for everyone. Here the misses come from that added noise; in real data they come from everything a model leaves out. The line is the least-squares line for these 11 points, and it stays fixed. In step 5 the ringed point can be dragged until its miss is 4 times its starting size, above or below the line. The stage shows values to two decimal places; the calculations use unrounded values.
If this helped, you can .