ML Explorer · Level 2 · Learning from data
Linear regression
Least squares chooses the straight line with the smallest total squared error.
Overview
Linear regression fits a straight line to points, so that the line can predict y from x. Lesson 1 showed a line like this being trained. This lesson opens it up.
You will place a line by eye first, and see its misses as squares. Then the least-squares formula finds the line with the smallest total area of squares. Lesson 1 promised to derive that formula, and Lesson 4 said it exists. Here it is.
The points are made up. See "About the data" at the end.
You'll learn
- what least squares makes as small as possible, and how the best line is found
- what the slope and the intercept of the fitted line mean, and where they stop meaning anything
- what R² measures, and what it cannot tell you
-
Which line?
These 25 points rise from left to right, roughly along a straight line. Which straight line describes that trend best?
Each dot is one example, with an input x and a known answer y.
The dashed lines are three candidates. Each passes through the middle of the cloud, and each looks reasonable. Looking is not enough to choose between them.
To pick one, we need a rule that says what "best" means. Lesson 4 gave us one: make the squared misses small. This lesson uses it.
A straight line is a simplification. It keeps the overall trend and ignores the scatter around it.
The maths
The data are pairs for , with . A straight line predicts
where is the intercept and the slope. Choosing a line means choosing these two numbers.
The means of the data are = and = . The three candidates all pass through the point .
-
Your line
Drag the two round handles up or down to place a line by eye. Two points fix a straight line, so the handles set it completely.
Each grey stick is a residual: the real y minus the line's prediction, as in Lesson 4.
The box shows the sum of squared errors, or SSE: square each residual, then add them up. Smaller means a closer fit. Try to make it as small as you can.
The starting line slopes the wrong way on purpose. Its SSE is .
To use the keyboard, Tab to a handle and use the arrow keys. Page Up and Page Down make bigger moves. "Reset my line" puts the line back where it started.
The maths
The handles sit at and . With heights and , your line has
Its residuals are , and
Lesson 4's mean squared error is . Dividing by the fixed number does not change which line is best, so this lesson works with SSE.
-
The squares are what we shrink
Each squared miss is now drawn as a square on its stick. Least squares means making the total area of all the squares as small as possible.
Each square's side is a miss, so its area is that miss squared. The SSE is the total area.
Keep dragging the handles. A line that gets close to most points but far from a few still has big squares at those few. Squaring makes big misses count much more than small ones.
The squares are drawn using the y-axis scale on both sides, so they look square on screen. That is a drawing choice; only their areas matter.
Finding the smallest total by dragging is slow, and you can never be sure you have it. The next step finds it exactly.
The maths
The box shows and , here with = .
is a function of two inputs, the intercept and the slope. Every pair gives one line and one total area. Lesson 5 showed that the bottom of a bowl is where the slope is zero. With two inputs, that means both partial derivatives are zero at once. Step 4 uses exactly that.
-
Solve it
Now let the formula choose. The line slides from yours to the least-squares line: the one straight line with the smallest possible SSE.
Watch the squares shrink as the line moves. Your line stays as a dashed line for comparison.
The bars compare the two SSEs. Whatever line you made, its SSE is at least as big as the least-squares SSE. It is equal only if you found the least-squares line exactly.
The least-squares line has intercept = and slope = , with SSE = . The hats mark numbers estimated from the data.
No searching is needed. For one input, two short formulas give the answer directly, using only the means, the spread of x, and how x and y vary together. The maths below derives them. Lesson 7 homes in on the same line another way, by walking downhill.
"Solve again" replays the move from your line.
The maths
We want the that makes smallest. At the bottom, both partial derivatives (Lesson 5) are zero. Holding fixed and changing :
Divide by : . So the best line passes through , and
Holding fixed and changing :
Put into it and divide by :
The deviations add up to zero, and so do the . So subtracting from the in front changes nothing, and the equation becomes , where
Hence the closed form, the formula from Lesson 1:
This needs , that is, x must have some spread. Then this flat point is the one and only minimum: the second derivatives are , and (mixed), and
Both and this number are positive, so SSE is a bowl that opens upwards in every direction.
Computing the least-squares line needs only that x has some spread. Inference (confidence intervals, tests) needs linearity, independent errors and constant variance, plus normal errors for exact small-sample results.
-
Reading the line
The slope says how much the prediction changes, on average, for each extra unit of x. The intercept is the prediction at x = 0, which may mean nothing if x = 0 lies outside the data.
Move the slider. Going one unit to the right along x always raises the prediction by the same amount, the slope, because the line is straight.
Here each extra unit of x adds to the prediction. It is an average change in the prediction, not a promise about any one point: the points scatter around the line.
The shaded band is the range of x the data cover. Inside it, the line is backed by data. The x axis runs a little past the data on both sides, and there the line is dashed: it still gives numbers, but nothing checks them. Using a line outside its data is called extrapolation, and it can go badly wrong if the real pattern bends.
The intercept is where the line meets x = 0 (the square). Here the smallest x is , so x = 0 lies just outside the data and â = is itself an extrapolation, though only a short one. In many real problems x = 0 is impossible or far from any data, and there the intercept has no real-world meaning. The line still needs it, to sit at the right height.
The slope describes how y and x go together in these data. On its own, it does not show that changing x would change y.
The maths
This holds for every , but the data only support it for between the smallest and largest . The intercept is .
These points were made with intercept 4 and slope 1.5 plus random noise. Least squares found = and = : close, but not exact, because the noise moves the estimates.
-
R²: how much of the spread the line explains
R² compares the line's squared misses with the squared misses of the flat line at the mean of y. It is the share of the variation in y that the line accounts for.
Press "Misses from the mean". The dashed sticks run to the flat line at the mean of y, the best constant guess from Lesson 4. Their squares add up to SST, the total sum of squares. It measures how much y varies before we use x at all.
Press "Misses from the line". These sticks are the residuals again, and their squares add up to the SSE.
R² = 1 − SSE/SST. If the line removes most of the misses, SSE is small next to SST and R² is near 1.
Here SST = and SSE = , so R² = . The line accounts for about % of the variation in y.
For the least-squares line with an intercept, R² is between 0 and 1 on the data it was fitted to (as long as y varies). The flat line at the mean is one of the lines least squares could choose, so its SSE can never be bigger than SST.
R² has limits:
- It does not show cause and effect.
- It does not show that a straight line is the right shape. Points on a gentle curve can still give a high R².
- It is computed on the same data the line was fitted to, so it says how well the line fits these points, not how well it will predict new ones. Lesson 9 deals with that.
The maths
is the population variance of y (divide by ). is the MSE, which is also the population variance of the residuals, because least-squares residuals average zero. The s cancel, so is one minus the share of the variance of y left in the residuals.
Putting and into the SSE gives, with ,
Here is the correlation of x and y, a number between −1 and 1 that measures how closely the points follow a straight line. holds only for simple regression (one input, a straight line with an intercept, fitted by least squares). With several inputs, or for any other line, it does not.
Here = , and its square is = .
Quick check
Lesson complete
- Least squares chooses the line with the smallest sum of squared residuals, the SSE. No other straight line does better on these data.
- For one input it has a closed form, and , found by setting both partial derivatives of the SSE to zero.
- The slope is the average change in the prediction per unit of x within the data. R² = 1 − SSE/SST is the share of the variation in y the line accounts for.
Common misconceptions
- "Least squares minimises the distance from each point to the line." It minimises squared vertical distances, the misses in y.
- "A high R² proves the line is right." It does not show cause and effect, and a curved pattern can still give a high R².
- "The intercept always means something." It is the prediction at x = 0, which may be outside the data or impossible.
- "Least squares only works if the data are normal." Computing the line needs only that x has spread. Assumptions such as normal errors matter for exact confidence intervals and tests in small samples.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 3.1.1, "Estimating the Coefficients", and Section 3.1.3, "Assessing the Accuracy of the Model", which covers the R² statistic and its link to the correlation in simple linear regression.
- Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 9.2, "Parameter Estimation", which derives the least-squares solution for linear regression with matrices.
About the data. The 25 points are made up. They come from a seeded random number generator, mulberry32 with seed 11. For each point in turn, x is drawn uniformly between 0 and 10, and then y = 4 + 1.5x plus 2 times a random draw from a normal distribution with mean 0 and SD 1. So the points are the same for everyone. Here the misses come from that added noise; in real data they come from everything a model leaves out. Your line's handles sit at x = 1 and x = 9 and move between 0 and 24. The stage shows values to two decimal places; the calculations use unrounded values.
If this helped, you can .