Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 1 · Data & intuition

Centre and spread

Two numbers can sum up a set of values: where they sit, and how far they spread.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

Here are nine commute times: how many minutes each of nine people took to get to work one morning. Nine numbers are hard to take in at once. Two summaries help: where the values sit, called their centre, and how far apart they are, called their spread.

You will meet two measures of centre, the mean and the median, and two measures of spread, the variance and the standard deviation. You can drag any of the nine values along the line and watch every number change.

The times are made up. See "About the data" at the end.

You'll learn

  • how the mean and the median measure the centre, and why one far-out value moves the mean but not the median
  • how the variance and the standard deviation measure spread, and when to divide by n − 1 instead of n
  • what a z-score tells you about a value
  1. The mean is a balance point

    Add up the values and divide by how many there are. That is the mean.

    The mean is what most people call the average.

    Each dot is one person's commute time, placed along a line from 0 to 120 minutes. Dots with nearly the same time are stacked so you can see them all.

    Picture each dot as an equal weight on a plank. The plank balances on the mean. This is a picture, not physics: the plank weighs nothing and only the dots count.

    It balances because the distances of the values below the mean add up to the same total as the distances of the values above it. The box shows both totals.

    Drag any dot, or use the keyboard: Tab moves between the nine dots, and the arrow keys move the selected one. Watch the balance point move. Times are whole minutes; the box rounds to one decimal place.

    The maths

    Write nn for the number of values and xix_i for the ii-th value. Here there is one feature, so one subscript is enough, and n=9n = 9. The mean, read "x-bar", is

    xˉ=1n∑i=1nxi\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i

    The distances from the mean, counted negative below it and positive above it, always add to zero:

    ∑i=1n(xi−xˉ)=∑i=1nxi−nxˉ=0\sum_{i=1}^{n} (x_i - \bar{x}) = \sum_{i=1}^{n} x_i - n\bar{x} = 0

    That is the balance.

  2. The median is the middle value

    Sort the values from smallest to largest. The median is the one in the middle.

    With nine values, the middle one is the 5th: four values are at or below it and four at or above it. The box lists all nine in order.

    The dashed line marks the median, and the ring shows which dot is the middle one.

    With an even number of values there would be two in the middle, and the median would be halfway between them. Here there are always nine, so that never comes up.

    The maths

    Write the values in sorted order as x(1)≤x(2)≤⋯≤x(n)x_{(1)} \le x_{(2)} \le \dots \le x_{(n)}. The bracketed subscript is the position in the sorted list. For odd nn, the median is x((n+1)/2)x_{((n+1)/2)}, which here is x(5)x_{(5)}. For even nn, it is 12(x(n/2)+x(n/2+1))\tfrac{1}{2}\big(x_{(n/2)} + x_{(n/2+1)}\big).

  3. One far-out value

    Drag the largest value far to the right. The mean follows it, and the median stays where it is.

    A value far from the rest is called an outlier. Imagine one person's train broke down that morning.

    The mean uses the size of every value. Moving one value by 9 minutes moves the mean by 1 minute, because there are 9 values.

    The median is just the middle value of the sorted list. The largest value stays the largest however far it goes, so the middle value does not change.

    Drag a value across the middle and the median can change, but only a little: it stays between the 4th and 6th values of the sorted list from before the move. Faint dotted lines mark where the mean and median started.

    The button moves the largest value to 120 for you. Reset puts every value back.

    The maths

    Moving one value by dd changes the sum by dd, so the mean changes by d/nd/n.

    The median is x(5)x_{(5)}. Moving x(9)x_{(9)} further up keeps the order, so x(5)x_{(5)} stays the same. Moving any one value anywhere leaves the new median between the old x(4)x_{(4)} and x(6)x_{(6)}.

    To drag the median as far as you like, you would have to move at least 5 of the 9 values. The mean needs only one. On this stage no value can pass 120 minutes, so even the mean has a limit here.

  4. Variance: the average squared distance

    To measure spread, take each value's distance from the mean, square it, and average the squares. That average is the variance.

    Squaring a distance gives the area of a square with that side. Each faint square runs from a dot across to the mean, so its area is that dot's squared distance.

    A value twice as far from the mean gets a square four times as big. Drag a dot far out and watch its square grow. A very large square runs off the top of the stage.

    The bright square, standing on the mean, has the average area of the nine. Its area is the variance.

    Squares are never negative, so values below the mean cannot cancel values above it, as the distances did in step 1.

    Variance is in squared minutes, which is hard to picture. Step 6 deals with that. Here we divide by n, the number of values; the next step explains the other choice.

    The maths

    The average squared distance from the mean is

    1n∑i=1n(xi−xˉ)2\frac{1}{n}\sum_{i=1}^{n} (x_i - \bar{x})^2

    If these nine values are the whole group you care about, the population, this is the population variance, written σ2\sigma^2, and their mean is the population mean, written μ\mu. Lesson 2 used the same symbols, with a subscript jj for each feature.

  5. Divide by n or by n − 1?

    If the nine values are a sample from a bigger group and you want the spread of that whole group, divide by n − 1 instead of n.

    Why? Distances measured from the sample's own mean come out too small on average, and dividing by n − 1 instead of n corrects that exactly for the variance. The maths shows how.

    The solid square divides by n = 9. The dashed square divides by n − 1 = 8, so it is bigger, unless all nine values are the same. The shaded strip is the extra area, and the box gives the ratio.

    Drag a dot far out: both squares grow, and the gap with them.

    Dividing by n gives the population variance: the spread of exactly these nine values. Dividing by n − 1 gives the sample variance: an estimate of the spread of the bigger group the nine came from, for example everyone who commutes in the same city.

    With many values, the two hardly differ. Every page in the Explorer says which one it uses, and the box here shows both.

    The maths

    Suppose the nine values are a sample: independent draws from a much larger population with mean μ\mu and variance σ2\sigma^2. Both are unknown. We estimate μ\mu with xˉ\bar{x}, and σ2\sigma^2 with the sample variance

    s2=1n−1∑i=1n(xi−xˉ)2s^2 = \frac{1}{n-1}\sum_{i=1}^{n} (x_i - \bar{x})^2

    Why n−1n - 1? The spread we want is measured from μ\mu, but we can only measure from xˉ\bar{x}. Of all numbers cc, it is c=xˉc = \bar{x} that makes ∑i(xi−c)2\sum_i (x_i - c)^2 smallest, so distances from xˉ\bar{x} come out too small on average. This identity says by how much:

    ∑i=1n(xi−μ)2=∑i=1n(xi−xˉ)2+n(xˉ−μ)2\begin{aligned} \sum_{i=1}^{n} (x_i - \mu)^2 &= \sum_{i=1}^{n} (x_i - \bar{x})^2 \\ &\quad + n(\bar{x} - \mu)^2 \end{aligned}

    Write E[ ⋅ ]\mathbb{E}[\,\cdot\,] for the average over many repeated samples, the expected value. On the left, each (xi−μ)2(x_i - \mu)^2 has expected value σ2\sigma^2, so the sum has nσ2n\sigma^2. The mean of nn independent draws varies less than one draw: E[(xˉ−μ)2]=σ2/n\mathbb{E}[(\bar{x} - \mu)^2] = \sigma^2/n, so the last term has expected value σ2\sigma^2. That leaves

    E[∑i=1n(xi−xˉ)2]=(n−1) σ2\mathbb{E}\Big[\sum_{i=1}^{n} (x_i - \bar{x})^2\Big] = (n - 1)\,\sigma^2

    and dividing by n−1n - 1 gives

    E[s2]=σ2\mathbb{E}[s^2] = \sigma^2

    Dividing by n−1n - 1 is called Bessel's correction. It makes s2s^2 an unbiased estimator of σ2\sigma^2: right on average over many samples, though any one sample can be too high or too low.

    Lesson 2 divided by nn because it described the six fruit themselves, not a bigger group they were drawn from.

  6. Standard deviation and z-scores

    Take the square root of the variance and you are back in minutes. That is the standard deviation, or SD: roughly, a typical distance from the mean.

    "Typical distance" is a simplification. The SD is the square root of the average squared distance, so big distances count for more than they would in a plain average.

    The ruler above the dots is marked in SDs from the mean. A value's z-score says how many SDs it sits from the mean: positive above the mean, negative below. A z-score of −2 means two SDs below the mean.

    Which SD? The population SD (divide by n) describes these nine values themselves. The sample SD (divide by n − 1) estimates the spread of the bigger group they came from. For z-scores this page uses the population SD, a common convention when standardising and the one Lesson 2 used.

    Drag or select a dot to see its z-score in the box. The box rounds z-scores to two decimal places.

    The maths

    The two SDs are the square roots of the two variances: σ=σ2\sigma = \sqrt{\sigma^2} divides by nn, and s=s2s = \sqrt{s^2} divides by n−1n - 1.

    z-scores here treat the nine values as the population, as Lesson 2 did, so μ=xˉ\mu = \bar{x} and

    zi=xi−μσz_i = \frac{x_i - \mu}{\sigma}

    So zi=−2z_i = -2 means xi=μ−2σx_i = \mu - 2\sigma. If every value were the same, σ\sigma would be 0 and the z-score would not be defined. Using ss instead would multiply every z-score by (n−1)/n\sqrt{(n-1)/n}, about 0.94 when n=9n = 9.

    The square root undoes the unbiasedness of step 5. The sample SD ss is a biased estimator of σ\sigma: on average it comes out below σ\sigma, though the gap shrinks as nn grows.

Quick check

One of the nine values is moved much further out. Which of these resists that change best?

The mean uses the size of every value, so one far value pulls it along. Moving one of nine values by 9 minutes moves the mean by 1 minute.

Right. The median is the middle value of the sorted list. Moving an end value further out keeps the order, so the middle value stays put.

The variance squares each distance from the mean, so one far value adds a very large square and the variance jumps.

The SD is the square root of the variance, so one far value raises it as well.

A commute time has a z-score of −2. What does that mean?

A z-score counts standard deviations, not minutes. Multiply it by the SD to get the distance from the mean in minutes.

The sign compares the value with the mean, not with zero. A negative z-score means below the mean; the time itself is still positive.

Right. The z-score is the distance from the mean divided by the SD, so −2 means two SDs below the mean.

A z-score only says where a value sits. A value far from the mean is worth checking, but it may be perfectly real.

Lesson complete

  • The mean is the balance point of the values. The median is the middle value once they are sorted.
  • One far-out value pulls the mean along but barely moves the median.
  • The variance is the average squared distance from the mean. The SD is its square root, back in the original units.

Common misconceptions

  • "The mean is always the typical value." When a few values sit far out on one side, the median can describe a typical value better.
  • "An outlier is a mistake, so delete it." Check it first. It may be an error, or it may be the most interesting value you have.
  • "Always divide by n − 1." Divide by n − 1 to estimate the spread of a bigger group from a sample. Divide by n to describe the values you have. Either way, say which.
  • "Since the sample variance is unbiased, so is the sample SD." Taking the square root brings a bias back: on average the sample SD comes out below the population SD.
Next lesson: Measuring error →

Further reading

  1. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 6.4, "Summary Statistics and Independence", which defines means and variances, including the empirical (sample) versions, and notes the unbiased version with N−1N - 1 in the denominator.
  2. Goodfellow, Bengio & Courville, Deep Learning, Section 5.4, "Estimators, Bias and Variance", which works through why dividing by one less than the sample size gives an unbiased variance, and notes that the square root still underestimates the standard deviation.

About the data. The nine commute times are made up. They come from a seeded random number generator, mulberry32 with seed 21. Each time is 50 + 8 times a random draw from a normal distribution with mean 0 and SD 1, rounded to a whole minute. So the starting values are the same for everyone. Dragged values stay between 0 and 120 minutes and are rounded to whole minutes. The stage shows means, variances and SDs to one decimal place and z-scores to two; the calculations use unrounded values. Real commute times need not follow this bell-shaped pattern.

If this helped, you can .