Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 1 · Data & intuition

Measuring error

Every model misses. A loss sums up all the misses in one number.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

A model's predictions miss. To say how good a model is, or to train one, we need a single number that sums up how badly it misses. That number is called a loss. Smaller is better.

This lesson takes 11 points and one fixed line, and builds two losses from the line's misses: the mean absolute error (MAE) and the mean squared error (MSE). Lesson 1 promised to explain why we square. This is where it does.

The points are made up. See "About the data" at the end.

You'll learn

  • what a residual is, and why residuals can't simply be added up
  • how MAE and MSE are built, and how each one treats a single big miss
  • why squaring is the usual choice, and which single guess each loss prefers
  1. Residuals: how far each point misses

    A residual is a miss: the real y minus the line's prediction.

    Each vertical stick shows one residual, from a point to the line.

    The line is the least-squares line for these 11 points, the "best fit" from Lesson 1. It stays fixed for the whole lesson. We are measuring its misses, not moving it.

    For each point, the line gives a prediction, written ŷ and said "y-hat". The residual is y − ŷ. It is positive when the point sits above the line and negative when it sits below.

    The ringed point sits above the line. The box gives its y, its ŷ and its residual, e.

    The stage rounds to two decimal places.

    The maths

    The points are (xi,yi)(x_i, y_i) for i=1,…,ni = 1, \dots, n, with n=11n = 11. The fixed line is y^=a^+b^ x\hat{y} = \hat{a} + \hat{b}\,x, so point ii has prediction y^i=a^+b^ xi\hat{y}_i = \hat{a} + \hat{b}\,x_i and residual

    ei=yi−y^ie_i = y_i - \hat{y}_i

    The sign matters: ei>0e_i > 0 means the line predicted too low, and ei<0e_i < 0 means it predicted too high.

  2. Why not just add them up?

    Adding the residuals sounds like a fair total. But misses above the line are positive and misses below are negative, so they cancel.

    The chart on the right lays the sticks end to end. The ones above the line go up (solid). Then the ones below come back down (dashed).

    The walk ends where it started: the residuals add up to zero, yet every point misses.

    That is no accident. For a least-squares line like this one, the residuals always add up to zero. So the plain sum tells you nothing about how well the line fits.

    We need a total in which every miss counts against the line, whatever its sign. There are two common fixes: use the size of each miss, or its square.

    The maths

    For any line y^=a+b x\hat{y} = a + b\,x, with xˉ\bar{x} and yˉ\bar{y} the means of the xix_i and the yiy_i,

    ∑i=1nei=∑i=1n(yi−a−b xi)=n (yˉ−a−b xˉ)\begin{aligned} \sum_{i=1}^{n} e_i &= \sum_{i=1}^{n} (y_i - a - b\,x_i) \\ &= n\,(\bar{y} - a - b\,\bar{x}) \end{aligned}

    This is zero for every line that passes through the point (xˉ,yˉ)(\bar{x}, \bar{y}), however badly it is tilted. Lesson 1's formula a^=yˉ−b^ xˉ\hat{a} = \bar{y} - \hat{b}\,\bar{x} puts the least-squares line through that point, so its residuals add up to zero.

  3. MAE: the average size of a miss

    Drop the sign and keep the size. Average the sizes of the misses, and you have the mean absolute error, or MAE.

    The size of a number, ignoring its sign, is its absolute value, written |e|. So the absolute value of −2 is 2, and so is the absolute value of 2.

    The chart on the right stands every stick upright, in the order the points sit from left to right. The dashed bars hung below the line.

    The dashed line across the chart is the average height of the bars. That is the MAE.

    Every miss now counts against the line, and a miss twice as big counts twice as much.

    The maths

    The absolute value is ∣e∣=e|e| = e when e≥0e \ge 0, and ∣e∣=−e|e| = -e when e<0e < 0. Then

    MAE=1n∑i=1n∣ei∣=1n∑i=1n∣yi−y^i∣\begin{aligned} \mathrm{MAE} &= \frac{1}{n}\sum_{i=1}^{n} |e_i| \\ &= \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i| \end{aligned}
  4. MSE: square each miss

    Or square each miss and average the squares. That is the mean squared error, or MSE: the "average squared miss" from Lesson 1.

    Squaring a miss gives the area of a square with that side. Each stick now carries its square.

    On the right, the bars change from each |e| to each e². The big misses grow a lot. Misses smaller than 1 shrink: 0.5 squared is 0.25.

    Why square? Two reasons show here. A square is never negative, so misses cannot cancel. And big misses count for far more than small ones: a miss twice as big gets a square four times as big. Step 6 adds three more reasons.

    MSE is in squared units of y, which is hard to picture. Step 6 shows how to get back to the units of y.

    The maths
    MSE=1n∑i=1nei2=1n∑i=1n(yi−y^i)2\begin{aligned} \mathrm{MSE} &= \frac{1}{n}\sum_{i=1}^{n} e_i^2 \\ &= \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \end{aligned}

    The 1/n1/n makes it an average, so MSE does not grow just because there are more points.

    Which misses count as "smaller than 1" depends on the units of y: measure in centimetres instead of metres and every miss is 100 times bigger. So that detail is not deep. What does not depend on units is that squaring widens the gap between small and big misses: if one miss is rr times another, its square is r2r^2 times the other's square.

  5. One big miss

    Drag the ringed point away from the line. Its square grows much faster than its stick.

    The line stays where it is. Only this point moves, so only its residual changes.

    When its miss becomes k times as big, its part of the MAE becomes k times as big too: it grows in proportion. Its part of the MSE becomes k² times as big: 3 times the miss, 9 times the square.

    The chart on the right shows both. The dashed line is k and the solid curve is k².

    The MSE as a whole grows by less than nine times, because the other ten squares stay the same. The box shows both losses, before and after.

    This is what "MSE punishes big misses" means: one far-off point can dominate it. Whether that is what you want depends on the problem.

    Drag the point, or Tab to it and use the arrow keys. "Put it back" returns it, and it also goes back when you leave this step.

    The maths

    Split off the ringed point, number jj:

    MAE=1n(∣ej∣+∑i≠j∣ei∣)\mathrm{MAE} = \frac{1}{n}\Big(|e_j| + \sum_{i \ne j} |e_i|\Big)
    MSE=1n(ej2+∑i≠jei2)\mathrm{MSE} = \frac{1}{n}\Big(e_j^2 + \sum_{i \ne j} e_i^2\Big)

    The line is fixed, so dragging point jj changes only eje_j. Replace eje_j by k ejk\,e_j, with k≥0k \ge 0. Its part of the MAE goes from ∣ej∣/n|e_j|/n to k ∣ej∣/nk\,|e_j|/n. Its part of the MSE goes from ej2/ne_j^2/n to k2ej2/nk^2 e_j^2/n.

    The MAE rises in a straight line as kk grows. The MSE rises along a parabola. The other terms do not change, so the whole MSE grows by less than k2k^2 times when k>1k > 1.

    Dragging the point across the line flips the sign of eje_j. The stage counts the size, k=∣ej∣/∣ejstart∣k = |e_j| / |e_j^{\text{start}}|, and stops at k=4k = 4.

  6. The best constant guess

    Suppose you had to guess one number, c, for every point, ignoring x. Which c is best? It depends on the loss: the mean of y for MSE, the median of y for MAE.

    The line is now flat at height c, and each stick is a point's miss from c. Move the slider and watch the two curves on the right. Each curve gives the loss for every possible c, and the dots mark the current one.

    There are 11 points, an odd number, so the median is a single middle value and the lowest MAE is at that one c. With an even number of points, every c between the two middle values would give the same lowest MAE.

    The chart shows the RMSE, the square root of the MSE, so that both curves are in the units of y. The MSE itself is in squared units. The RMSE is lowest exactly where the MSE is.

    Look at the shapes. The MSE curve is smooth. The MAE curve has a corner at each value of y.

    So why square? It stops misses cancelling, and it punishes big misses more. It is smooth, so it has a slope everywhere, which makes it easy to follow downhill with gradient descent (Lesson 7). It leads to a formula for the best line (Lesson 6). And its best constant guess is the familiar mean.

    MAE has its own strength: one far-out point sways it less, just as one far-out value moves the median less than the mean.

    The maths

    For a constant guess, y^i=c\hat{y}_i = c for every ii. Write yˉ\bar{y} for the mean of the yiy_i. The MSE splits into two parts:

    1n∑i=1n(yi−c)2=1n∑i=1n(yi−yˉ)2+(yˉ−c)2\begin{aligned} &\frac{1}{n}\sum_{i=1}^{n} (y_i - c)^2 \\ &\quad = \frac{1}{n}\sum_{i=1}^{n} (y_i - \bar{y})^2 + (\bar{y} - c)^2 \end{aligned}

    The first part does not depend on cc, and the second is zero only at c=yˉc = \bar{y}. So the mean is the one best constant under MSE, and the lowest MSE is the population variance of y (divide by nn). Lesson 3 used the same fact when it explained n−1n - 1.

    For MAE, take cc between two neighbouring values of y, with pp values below it and qq above. Raising cc a little brings it closer to the qq values above and further from the pp below, so the slope of the MAE is (p−q)/n(p - q)/n. It is negative while fewer than half the values lie below cc and positive once more than half do. With nn odd, the MAE is therefore lowest at the middle value, the median, and nowhere else. With nn even, the slope is 0 between the two middle values, and every cc there ties. The slope jumps at each value of y, which is why the curve has corners.

    Since RMSE=MSE\mathrm{RMSE} = \sqrt{\mathrm{MSE}} and the square root only ever increases, the RMSE and the MSE are lowest at the same cc.

    One more reason for squaring comes from probability. Suppose yi=a+b xi+εiy_i = a + b\,x_i + \varepsilon_i, where the εi\varepsilon_i are independent draws from a normal distribution with mean 0 and a fixed SD. The likelihood is the probability density of the observed yiy_i, seen as a function of aa and bb. Its logarithm is a constant minus a positive multiple of ∑i(yi−a−b xi)2\sum_i (y_i - a - b\,x_i)^2, so the line with the highest likelihood is the least-squares line. These points were made exactly that way (see About the data).

Quick check

Which of these punishes one big miss most heavily?

MAE counts a miss by its size. Make one miss ten times as big and its part of the MAE is ten times as big, no more.

In the plain sum, a big miss above the line can be cancelled by misses below it, so it may not count against the line at all.

Right. MSE squares each miss, so one miss ten times as big gives a square a hundred times as big.

They differ. A miss's part of the MAE grows in proportion to its size, its part of the MSE grows with the square, and in the plain sum it can be cancelled out.

You must guess one number, c, for all 11 points. Which c gives the lowest MAE?

The mean gives the lowest MSE. Here the mean and the median differ, and the MAE is lowest at the median.

Halfway between the extremes looks at only two values. It makes the biggest single miss as small as possible, not the average miss.

With these 11 points, zero would be best only if the median were zero. Here every y is above zero, so c = 0 misses every point by its whole value.

Right. Moving c towards the middle brings it closer to more points than it moves away from, so the MAE falls until c reaches the middle value. With 11 points that is a single value, the median.

Lesson complete

  • A residual is a miss, y − ŷ. Positive and negative residuals cancel, so their plain sum hides how badly a line fits.
  • MAE averages the sizes of the misses. MSE averages their squares, so one big miss counts for much more.
  • For a single constant guess, MSE is lowest at the mean and MAE at the median.

Common misconceptions

  • "Residuals that add up to zero mean a perfect fit." The least-squares line always makes them add up to zero, however big the misses are.
  • "MSE is in the same units as y." It is in squared units. Its square root, the RMSE, is in the units of y.
  • "MAE and MSE always agree on the best answer." For a single constant guess, MSE picks the mean and MAE picks the median, and these often differ.
  • "Squaring is the only correct way to measure error." It is a choice, with good reasons behind it. When a few far-off points should not dominate, MAE can suit better.
Next lesson: Slopes and derivatives →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 2.2.1, "Measuring the Quality of Fit", which introduces the MSE as the most common measure of fit in regression.
  2. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 9.2, "Parameter Estimation", which shows how maximum likelihood with normal noise leads to the squared-error loss, and notes that the RMSE has the same units as y.
  3. Goodfellow, Bengio & Courville, Deep Learning, Section 5.5.1, "Conditional Log-Likelihood and Mean Squared Error", which links the mean squared error to maximum likelihood for linear regression.

About the data. The 11 points are made up. They come from a seeded random number generator, mulberry32 with seed 5. For each point in turn, x is drawn uniformly between 0 and 10, and then y = 2 + 0.8x plus a random draw from a normal distribution with mean 0 and SD 1. So the points are the same for everyone. Here the misses come from that added noise; in real data they come from everything a model leaves out. The line is the least-squares line for these 11 points, and it stays fixed. In step 5 the ringed point can be dragged until its miss is 4 times its starting size, above or below the line. The stage shows values to two decimal places; the calculations use unrounded values.

If this helped, you can .