Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 2 · Learning from data

Linear regression

Least squares chooses the straight line with the smallest total squared error.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

Linear regression fits a straight line to points, so that the line can predict y from x. Lesson 1 showed a line like this being trained. This lesson opens it up.

You will place a line by eye first, and see its misses as squares. Then the least-squares formula finds the line with the smallest total area of squares. Lesson 1 promised to derive that formula, and Lesson 4 said it exists. Here it is.

The points are made up. See "About the data" at the end.

You'll learn

  • what least squares makes as small as possible, and how the best line is found
  • what the slope and the intercept of the fitted line mean, and where they stop meaning anything
  • what R² measures, and what it cannot tell you
  1. Which line?

    These 25 points rise from left to right, roughly along a straight line. Which straight line describes that trend best?

    Each dot is one example, with an input x and a known answer y.

    The dashed lines are three candidates. Each passes through the middle of the cloud, and each looks reasonable. Looking is not enough to choose between them.

    To pick one, we need a rule that says what "best" means. Lesson 4 gave us one: make the squared misses small. This lesson uses it.

    A straight line is a simplification. It keeps the overall trend and ignores the scatter around it.

    The maths

    The data are pairs (xi,yi)(x_i, y_i) for i=1,…,ni = 1, \dots, n, with n=25n = 25. A straight line predicts

    y^=a+b x\hat{y} = a + b\,x

    where aa is the intercept and bb the slope. Choosing a line means choosing these two numbers.

  2. Your line

    Drag the two round handles up or down to place a line by eye. Two points fix a straight line, so the handles set it completely.

    Each grey stick is a residual: the real y minus the line's prediction, as in Lesson 4.

    The box shows the sum of squared errors, or SSE: square each residual, then add them up. Smaller means a closer fit. Try to make it as small as you can.

    To use the keyboard, Tab to a handle and use the arrow keys. Page Up and Page Down make bigger moves. "Reset my line" puts the line back where it started.

    The maths

    The handles sit at x=1x = 1 and x=9x = 9. With heights h1h_1 and h2h_2, your line has

    b=h2−h19−1,a=h1−bb = \frac{h_2 - h_1}{9 - 1}, \qquad a = h_1 - b

    Its residuals are ei=yi−y^i=yi−a−b xie_i = y_i - \hat{y}_i = y_i - a - b\,x_i, and

    SSE(a,b)=∑i=1nei2=∑i=1n(yi−a−b xi)2\begin{aligned} \mathrm{SSE}(a, b) &= \sum_{i=1}^{n} e_i^2 \\ &= \sum_{i=1}^{n} \left(y_i - a - b\,x_i\right)^2 \end{aligned}

    Lesson 4's mean squared error is MSE=SSE/n\mathrm{MSE} = \mathrm{SSE}/n. Dividing by the fixed number nn does not change which line is best, so this lesson works with SSE.

  3. The squares are what we shrink

    Each squared miss is now drawn as a square on its stick. Least squares means making the total area of all the squares as small as possible.

    Each square's side is a miss, so its area is that miss squared. The SSE is the total area.

    Keep dragging the handles. A line that gets close to most points but far from a few still has big squares at those few. Squaring makes big misses count much more than small ones.

    The squares are drawn using the y-axis scale on both sides, so they look square on screen. That is a drawing choice; only their areas matter.

    Finding the smallest total by dragging is slow, and you can never be sure you have it. The next step finds it exactly.

    The maths

    SSE(a,b)\mathrm{SSE}(a, b) is a function of two inputs, the intercept and the slope. Every pair (a,b)(a, b) gives one line and one total area. Lesson 5 showed that the bottom of a bowl is where the slope is zero. With two inputs, that means both partial derivatives are zero at once. Step 4 uses exactly that.

  4. Solve it

    Now let the formula choose. The line slides from yours to the least-squares line: the one straight line with the smallest possible SSE.

    Watch the squares shrink as the line moves. Your line stays as a dashed line for comparison.

    The bars compare the two SSEs. Whatever line you made, its SSE is at least as big as the least-squares SSE. It is equal only if you found the least-squares line exactly.

    No searching is needed. For one input, two short formulas give the answer directly, using only the means, the spread of x, and how x and y vary together. The maths below derives them. Lesson 7 homes in on the same line another way, by walking downhill.

    "Solve again" replays the move from your line.

    The maths

    We want the (a,b)(a, b) that makes SSE(a,b)=∑i(yi−a−b xi)2\mathrm{SSE}(a, b) = \sum_i (y_i - a - b\,x_i)^2 smallest. At the bottom, both partial derivatives (Lesson 5) are zero. Holding bb fixed and changing aa:

    ∂ SSE∂a=−2∑i=1n(yi−a−b xi)=0\begin{aligned} \frac{\partial\,\mathrm{SSE}}{\partial a} &= -2\sum_{i=1}^{n} (y_i - a - b\,x_i) \\ &= 0 \end{aligned}

    Divide by −2n-2n: yˉ−a−b xˉ=0\bar{y} - a - b\,\bar{x} = 0. So the best line passes through (xˉ,yˉ)(\bar{x}, \bar{y}), and

    a^=yˉ−b^ xˉ\hat{a} = \bar{y} - \hat{b}\,\bar{x}

    Holding aa fixed and changing bb:

    ∂ SSE∂b=−2∑i=1nxi (yi−a−b xi)=0\begin{aligned} \frac{\partial\,\mathrm{SSE}}{\partial b} &= -2\sum_{i=1}^{n} x_i\,(y_i - a - b\,x_i) \\ &= 0 \end{aligned}

    Put a=yˉ−b xˉa = \bar{y} - b\,\bar{x} into it and divide by −2-2:

    ∑i=1nxi[(yi−yˉ)−b (xi−xˉ)]=0\begin{aligned} \sum_{i=1}^{n} x_i \big[&(y_i - \bar{y}) \\ &- b\,(x_i - \bar{x})\big] = 0 \end{aligned}

    The deviations yi−yˉy_i - \bar{y} add up to zero, and so do the xi−xˉx_i - \bar{x}. So subtracting xˉ\bar{x} from the xix_i in front changes nothing, and the equation becomes Sxy=b SxxS_{xy} = b\,S_{xx}, where

    Sxx=∑i=1n(xi−xˉ)2Sxy=∑i=1n(xi−xˉ)(yi−yˉ)\begin{aligned} S_{xx} &= \sum_{i=1}^{n} (x_i - \bar{x})^2 \\ S_{xy} &= \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) \end{aligned}

    Hence the closed form, the formula from Lesson 1:

    b^=SxySxxa^=yˉ−b^ xˉ\begin{aligned} \hat{b} &= \frac{S_{xy}}{S_{xx}} \\ \hat{a} &= \bar{y} - \hat{b}\,\bar{x} \end{aligned}

    This needs Sxx>0S_{xx} > 0, that is, x must have some spread. Then this flat point is the one and only minimum: the second derivatives are 2n2n, 2∑ixi22\sum_i x_i^2 and 2∑ixi2\sum_i x_i (mixed), and

    (2n)(2∑ixi2)−(2∑ixi)2=4n Sxx>0\begin{aligned} &(2n)\Big(2\sum_i x_i^2\Big) - \Big(2\sum_i x_i\Big)^2 \\ &\quad = 4n\,S_{xx} > 0 \end{aligned}

    Both 2n2n and this number are positive, so SSE is a bowl that opens upwards in every direction.

    Computing the least-squares line needs only that x has some spread. Inference (confidence intervals, tests) needs linearity, independent errors and constant variance, plus normal errors for exact small-sample results.

  5. Reading the line

    The slope says how much the prediction changes, on average, for each extra unit of x. The intercept is the prediction at x = 0, which may mean nothing if x = 0 lies outside the data.

    Move the slider. Going one unit to the right along x always raises the prediction by the same amount, the slope, because the line is straight.

    The shaded band is the range of x the data cover. Inside it, the line is backed by data. The x axis runs a little past the data on both sides, and there the line is dashed: it still gives numbers, but nothing checks them. Using a line outside its data is called extrapolation, and it can go badly wrong if the real pattern bends.

    The slope describes how y and x go together in these data. On its own, it does not show that changing x would change y.

    The maths
    y^(x+1)−y^(x)=(a^+b^ (x+1))−(a^+b^ x)=b^\begin{aligned} &\hat{y}(x + 1) - \hat{y}(x) \\ &\quad = \big(\hat{a} + \hat{b}\,(x + 1)\big) - \big(\hat{a} + \hat{b}\,x\big) \\ &\quad = \hat{b} \end{aligned}

    This holds for every xx, but the data only support it for xx between the smallest and largest xix_i. The intercept is y^(0)=a^\hat{y}(0) = \hat{a}.

  6. R²: how much of the spread the line explains

    R² compares the line's squared misses with the squared misses of the flat line at the mean of y. It is the share of the variation in y that the line accounts for.

    Press "Misses from the mean". The dashed sticks run to the flat line at the mean of y, the best constant guess from Lesson 4. Their squares add up to SST, the total sum of squares. It measures how much y varies before we use x at all.

    Press "Misses from the line". These sticks are the residuals again, and their squares add up to the SSE.

    R² = 1 − SSE/SST. If the line removes most of the misses, SSE is small next to SST and R² is near 1.

    For the least-squares line with an intercept, R² is between 0 and 1 on the data it was fitted to (as long as y varies). The flat line at the mean is one of the lines least squares could choose, so its SSE can never be bigger than SST.

    R² has limits:

    • It does not show cause and effect.
    • It does not show that a straight line is the right shape. Points on a gentle curve can still give a high R².
    • It is computed on the same data the line was fitted to, so it says how well the line fits these points, not how well it will predict new ones. Lesson 9 deals with that.
    The maths
    SST=∑i=1n(yi−yˉ)2R2=1−SSESST\begin{aligned} \mathrm{SST} &= \sum_{i=1}^{n} (y_i - \bar{y})^2 \\ R^2 &= 1 - \frac{\mathrm{SSE}}{\mathrm{SST}} \end{aligned}

    SST/n\mathrm{SST}/n is the population variance of y (divide by nn). SSE/n\mathrm{SSE}/n is the MSE, which is also the population variance of the residuals, because least-squares residuals average zero. The nns cancel, so R2R^2 is one minus the share of the variance of y left in the residuals.

    Putting a^\hat{a} and b^\hat{b} into the SSE gives, with Syy=SSTS_{yy} = \mathrm{SST},

    SSE=Syy−Sxy2SxxR2=Sxy2Sxx Syy=r2\begin{aligned} \mathrm{SSE} &= S_{yy} - \frac{S_{xy}^2}{S_{xx}} \\ R^2 &= \frac{S_{xy}^2}{S_{xx}\,S_{yy}} = r^2 \end{aligned}

    Here r=Sxy/Sxx Syyr = S_{xy}/\sqrt{S_{xx}\,S_{yy}} is the correlation of x and y, a number between −1 and 1 that measures how closely the points follow a straight line. R2=r2R^2 = r^2 holds only for simple regression (one input, a straight line with an intercept, fitted by least squares). With several inputs, or for any other line, it does not.

Quick check

What does least squares make as small as possible?

Residuals above and below the line cancel. Every line through the point where x and y are both at their means makes them add up to zero, however badly it fits, so this cannot pick one line.

Least squares measures each miss straight up or down, in the direction of y, because the line is used to predict y from x. Perpendicular distances lead to a different method, which usually gives a different line.

Right. The SSE is the sum of the squared residuals, the total area of the squares, and least squares finds the line with the smallest one.

With scattered data, no straight line passes through most of the points, and counting ignores how far off each point is. Least squares uses the size of every miss.

What does the slope of a least-squares line tell you?

That is the intercept. It may mean nothing when x = 0 lies outside the data.

Right. Each step of one unit along x moves the prediction by the slope. It describes the trend in the data, and outside their range nothing backs it up.

That would be a claim about cause and effect. A fitted slope only describes how y and x go together in these data.

That is R². A steep slope can come with a low R², and a gentle slope with a high one.

Lesson complete

  • Least squares chooses the line with the smallest sum of squared residuals, the SSE. No other straight line does better on these data.
  • For one input it has a closed form, b^=Sxy/Sxx\hat{b} = S_{xy}/S_{xx} and a^=yˉ−b^ xˉ\hat{a} = \bar{y} - \hat{b}\,\bar{x}, found by setting both partial derivatives of the SSE to zero.
  • The slope is the average change in the prediction per unit of x within the data. R² = 1 − SSE/SST is the share of the variation in y the line accounts for.

Common misconceptions

  • "Least squares minimises the distance from each point to the line." It minimises squared vertical distances, the misses in y.
  • "A high R² proves the line is right." It does not show cause and effect, and a curved pattern can still give a high R².
  • "The intercept always means something." It is the prediction at x = 0, which may be outside the data or impossible.
  • "Least squares only works if the data are normal." Computing the line needs only that x has spread. Assumptions such as normal errors matter for exact confidence intervals and tests in small samples.
Next lesson: Gradient descent →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 3.1.1, "Estimating the Coefficients", and Section 3.1.3, "Assessing the Accuracy of the Model", which covers the R² statistic and its link to the correlation in simple linear regression.
  2. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 9.2, "Parameter Estimation", which derives the least-squares solution for linear regression with matrices.

About the data. The 25 points are made up. They come from a seeded random number generator, mulberry32 with seed 11. For each point in turn, x is drawn uniformly between 0 and 10, and then y = 4 + 1.5x plus 2 times a random draw from a normal distribution with mean 0 and SD 1. So the points are the same for everyone. Here the misses come from that added noise; in real data they come from everything a model leaves out. Your line's handles sit at x = 1 and x = 9 and move between 0 and 24. The stage shows values to two decimal places; the calculations use unrounded values.

If this helped, you can .