ML Explorer · Level 1 · Data & intuition
Centre and spread
Two numbers can sum up a set of values: where they sit, and how far they spread.
Overview
Here are nine commute times: how many minutes each of nine people took to get to work one morning. Nine numbers are hard to take in at once. Two summaries help: where the values sit, called their centre, and how far apart they are, called their spread.
You will meet two measures of centre, the mean and the median, and two measures of spread, the variance and the standard deviation. You can drag any of the nine values along the line and watch every number change.
The times are made up. See "About the data" at the end.
You'll learn
- how the mean and the median measure the centre, and why one far-out value moves the mean but not the median
- how the variance and the standard deviation measure spread, and when to divide by n − 1 instead of n
- what a z-score tells you about a value
-
The mean is a balance point
Add up the values and divide by how many there are. That is the mean.
The mean is what most people call the average.
Each dot is one person's commute time, placed along a line from 0 to 120 minutes. Dots with nearly the same time are stacked so you can see them all.
Picture each dot as an equal weight on a plank. The plank balances on the mean. This is a picture, not physics: the plank weighs nothing and only the dots count.
It balances because the distances of the values below the mean add up to the same total as the distances of the values above it. The box shows both totals.
Drag any dot, or use the keyboard: Tab moves between the nine dots, and the arrow keys move the selected one. Watch the balance point move. Times are whole minutes; the box rounds to one decimal place.
The maths
Write for the number of values and for the -th value. Here there is one feature, so one subscript is enough, and . The mean, read "x-bar", is
The distances from the mean, counted negative below it and positive above it, always add to zero:
That is the balance.
For the starting data, the nine times add up to minutes, so = / = minutes, to one decimal place.
-
The median is the middle value
Sort the values from smallest to largest. The median is the one in the middle.
With nine values, the middle one is the 5th: four values are at or below it and four at or above it. The box lists all nine in order.
The dashed line marks the median, and the ring shows which dot is the middle one.
With an even number of values there would be two in the middle, and the median would be halfway between them. Here there are always nine, so that never comes up.
The maths
Write the values in sorted order as . The bracketed subscript is the position in the sorted list. For odd , the median is , which here is . For even , it is .
The starting data, sorted: . The 5th of these is .
-
One far-out value
Drag the largest value far to the right. The mean follows it, and the median stays where it is.
A value far from the rest is called an outlier. Imagine one person's train broke down that morning.
The mean uses the size of every value. Moving one value by 9 minutes moves the mean by 1 minute, because there are 9 values.
The median is just the middle value of the sorted list. The largest value stays the largest however far it goes, so the middle value does not change.
Moving the largest value from to minutes raises the mean from to . The median stays at .
Drag a value across the middle and the median can change, but only a little: it stays between the 4th and 6th values of the sorted list from before the move. Faint dotted lines mark where the mean and median started.
The button moves the largest value to 120 for you. Reset puts every value back.
The maths
Moving one value by changes the sum by , so the mean changes by .
The median is . Moving further up keeps the order, so stays the same. Moving any one value anywhere leaves the new median between the old and .
To drag the median as far as you like, you would have to move at least 5 of the 9 values. The mean needs only one. On this stage no value can pass 120 minutes, so even the mean has a limit here.
-
Variance: the average squared distance
To measure spread, take each value's distance from the mean, square it, and average the squares. That average is the variance.
Squaring a distance gives the area of a square with that side. Each faint square runs from a dot across to the mean, so its area is that dot's squared distance.
A value twice as far from the mean gets a square four times as big. Drag a dot far out and watch its square grow. A very large square runs off the top of the stage.
The bright square, standing on the mean, has the average area of the nine. Its area is the variance.
Squares are never negative, so values below the mean cannot cancel values above it, as the distances did in step 1.
Variance is in squared minutes, which is hard to picture. Step 6 deals with that. Here we divide by n, the number of values; the next step explains the other choice.
The maths
The average squared distance from the mean is
If these nine values are the whole group you care about, the population, this is the population variance, written , and their mean is the population mean, written . Lesson 2 used the same symbols, with a subscript for each feature.
For the starting data, the squared distances add up to min², so their average is / = min².
-
Divide by n or by n − 1?
If the nine values are a sample from a bigger group and you want the spread of that whole group, divide by n − 1 instead of n.
Why? Distances measured from the sample's own mean come out too small on average, and dividing by n − 1 instead of n corrects that exactly for the variance. The maths shows how.
The solid square divides by n = 9. The dashed square divides by n − 1 = 8, so it is bigger, unless all nine values are the same. The shaded strip is the extra area, and the box gives the ratio.
Drag a dot far out: both squares grow, and the gap with them.
Dividing by n gives the population variance: the spread of exactly these nine values. Dividing by n − 1 gives the sample variance: an estimate of the spread of the bigger group the nine came from, for example everyone who commutes in the same city.
With many values, the two hardly differ. Every page in the Explorer says which one it uses, and the box here shows both.
The maths
Suppose the nine values are a sample: independent draws from a much larger population with mean and variance . Both are unknown. We estimate with , and with the sample variance
Why ? The spread we want is measured from , but we can only measure from . Of all numbers , it is that makes smallest, so distances from come out too small on average. This identity says by how much:
Write for the average over many repeated samples, the expected value. On the left, each has expected value , so the sum has . The mean of independent draws varies less than one draw: , so the last term has expected value . That leaves
and dividing by gives
Dividing by is called Bessel's correction. It makes an unbiased estimator of : right on average over many samples, though any one sample can be too high or too low.
Lesson 2 divided by because it described the six fruit themselves, not a bigger group they were drawn from.
For the starting data, /8 = min².
-
Standard deviation and z-scores
Take the square root of the variance and you are back in minutes. That is the standard deviation, or SD: roughly, a typical distance from the mean.
"Typical distance" is a simplification. The SD is the square root of the average squared distance, so big distances count for more than they would in a plain average.
The ruler above the dots is marked in SDs from the mean. A value's z-score says how many SDs it sits from the mean: positive above the mean, negative below. A z-score of −2 means two SDs below the mean.
Which SD? The population SD (divide by n) describes these nine values themselves. The sample SD (divide by n − 1) estimates the spread of the bigger group they came from. For z-scores this page uses the population SD, a common convention when standardising and the one Lesson 2 used.
Drag or select a dot to see its z-score in the box. The box rounds z-scores to two decimal places.
The maths
The two SDs are the square roots of the two variances: divides by , and divides by .
z-scores here treat the nine values as the population, as Lesson 2 did, so and
So means . If every value were the same, would be 0 and the z-score would not be defined. Using instead would multiply every z-score by , about 0.94 when .
The square root undoes the unbiasedness of step 5. The sample SD is a biased estimator of : on average it comes out below , though the gap shrinks as grows.
For the starting data, the population SD is minutes and the sample SD is minutes.
Quick check
Lesson complete
- The mean is the balance point of the values. The median is the middle value once they are sorted.
- One far-out value pulls the mean along but barely moves the median.
- The variance is the average squared distance from the mean. The SD is its square root, back in the original units.
Common misconceptions
- "The mean is always the typical value." When a few values sit far out on one side, the median can describe a typical value better.
- "An outlier is a mistake, so delete it." Check it first. It may be an error, or it may be the most interesting value you have.
- "Always divide by n − 1." Divide by n − 1 to estimate the spread of a bigger group from a sample. Divide by n to describe the values you have. Either way, say which.
- "Since the sample variance is unbiased, so is the sample SD." Taking the square root brings a bias back: on average the sample SD comes out below the population SD.
Further reading
- Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 6.4, "Summary Statistics and Independence", which defines means and variances, including the empirical (sample) versions, and notes the unbiased version with in the denominator.
- Goodfellow, Bengio & Courville, Deep Learning, Section 5.4, "Estimators, Bias and Variance", which works through why dividing by one less than the sample size gives an unbiased variance, and notes that the square root still underestimates the standard deviation.
About the data. The nine commute times are made up. They come from a seeded random number generator, mulberry32 with seed 21. Each time is 50 + 8 times a random draw from a normal distribution with mean 0 and SD 1, rounded to a whole minute. So the starting values are the same for everyone. Dragged values stay between 0 and 120 minutes and are rounded to whole minutes. The stage shows means, variances and SDs to one decimal place and z-scores to two; the calculations use unrounded values. Real commute times need not follow this bell-shaped pattern.
If this helped, you can .