ML Explorer · Level 3 · Classic ML
Logistic regression
Squash a straight-line score into a probability, then pick a threshold to turn it into a class.
Overview
In Lesson 11, k-NN gave each new case a label by a vote. Logistic regression gives something more useful first: a probability. "This student has a 0.8 chance of passing." A threshold then turns that probability into a label.
We go back to students and study hours, as in Lesson 1, but now we only know whether each student passed. We see why a straight line is the wrong shape, bend it into an S, score it, fit it, and then decide where to draw the line between pass and fail.
The students are made up. See "About the data" at the end.
You'll learn
- why a straight line is the wrong shape for a probability, and how the sigmoid curve fixes it
- how log-loss scores a probability, and how fitting finds the best curve
- how a threshold turns a probability into a class, and what precision and recall measure
-
Why a straight line fails
Write a pass as 1 and a fail as 0. Fit a straight line to those 0s and 1s, and it soon predicts values no chance can take.
Each point is one student: hours studied across, the result up. A pass is y = 1, a violet triangle. A fail is y = 0, a cyan circle. Of the 30 students, 13 passed and 17 failed.
More hours go with more passes, but not neatly. Some students who studied little passed, and some who studied a lot failed.
The dashed line is the least-squares straight line through the 0s and 1s, fitted as in Lesson 6. We would like to read its height as the chance of passing.
It gives −0.19 at 0 hours and 1.17 at 10 hours. A chance cannot be below 0 or above 1. The line drops below 0 under 1.4 hours and rises above 1 past 8.7 hours.
We need a curve that stays between 0 and 1: low for few hours, high for many, and rising in between.
The maths
From this lesson on we write a straight line as
with the intercept and the slope, which machine learning calls a weight. Lesson 1 wrote , with as the slope; here has a new job.
Whatever the data, a line with leaves the range 0 to 1 once is far enough out, because keeps growing in one direction. On these data it already leaves inside the 0 to 10 hours we observed.
-
The sigmoid
Work out a straight-line score, then squash it between 0 and 1 with the sigmoid. The result is a probability.
For a student who studied x hours, the score is z = b + w·x, a straight line. The sigmoid σ turns any score into a number between 0 and 1, so the model is p̂ = σ(b + w·x). σ here is the sigmoid, not a standard deviation.
p̂, said "p-hat", is the predicted probability of class 1. Here that is the chance of passing.
Move the sliders. b slides the curve left and right. w sets how steep it is: a larger w gives a sharper change from likely fail to likely pass. The middle of the curve sits at −b ÷ w hours, so changing w moves the middle too, unless b changes with it. A negative w turns the curve round, so more hours would mean a lower chance.
The curve crosses 0.5 where the score is 0, at x = −b ÷ w hours. A large score gives p̂ close to 1 and a very negative one p̂ close to 0, but never exactly 0 or 1.
The maths
The sigmoid, also called the logistic function, is
σ here is the sigmoid, not a standard deviation. It has and , so the curve is symmetric about its middle.
Turn it inside out. The odds compare the chance of passing with the chance of failing, and their natural log, the log-odds, is the straight line:
So each extra hour adds to the log-odds, which multiplies the odds by . The model is linear in the log-odds, not in the probability.
-
Log-loss: scoring a probability
Score each student by the probability the curve gave to what really happened. A confident wrong answer costs the most.
The bars under the plot are each student's penalty. For a student who passed, it is −ln p̂. For one who failed, it is −ln(1 − p̂). ln is the natural logarithm.
If the curve gave what happened a probability near 1, the penalty is near 0. A probability of 0.5 costs 0.69, and 0.01 costs 4.6. As the probability goes to 0, the penalty grows without limit.
The mean of the penalties is the log-loss. Move the sliders and watch it change. The ringed bar is the largest penalty.
At the best curve, found in the next step, the largest penalty is 1.56. It belongs to the student who studied 6.30 hours and failed: the curve gave them a 0.79 chance of passing.
The maths
The mean log-loss over students is
For each student one of the two terms is zero, since is 0 or 1. With every and the log-loss is , about 0.693.
Making the log-loss small is the same as making the product of the probabilities given to what happened large, because is that product. Choosing the model this way is called maximum likelihood.
Why not the squared miss of Lesson 4? On top of the sigmoid it is not convex in and in general, and it never charges more than 1 for a wrong answer, however confident. Log-loss is convex, and it punishes confident mistakes hard.
The page clips to the range before taking the log, so every penalty stays finite. Bars taller than 3 are cut and marked with two short lines.
-
Fitting: walking downhill
Training finds the b and w with the smallest log-loss. Gradient descent gets there one small step at a time.
Press Fit. Gradient descent starts at b = 0 and w = 0, where the curve is flat at 0.5 and the log-loss is 0.693. Each step moves b and w a little way downhill.
The chart shows the log-loss after each step. It falls fast at first, then slowly. With learning rate η = 0.25 it takes 767 steps to come within 0.001 of the minimum, and after 3000 steps it is within 0.000001.
The dashed curve is the answer from Newton's method, a faster method that also uses how the slope of the loss changes. It needs 6 steps: b = −5.41, w = 1.07, log-loss 0.328. That curve crosses 0.5 at 5.06 hours.
There is no trap to fall into. The log-loss is convex: one bowl with no other dips, so walking downhill from any start ends at the same bottom.
The maths
The gradient of the log-loss has a simple form: the miss times the input.
Each step is and . Unlike least squares, setting the gradient to zero has no formula for the answer, so it is found step by step.
The second derivatives form the matrix
which is never negative in any direction, so the loss is convex. Newton's method multiplies the gradient by the inverse of this matrix (with one weight it would divide by the curvature); gradient descent uses only the gradient. Gradient descent is slow here because the hours are far from 0: they are not centred, so a change in w also moves the curve's middle, and b has to follow. That ties b and w together and gives the long, narrow valley of Lesson 8. With η = 0.25 every step lowers the loss on these data.
If some number of hours split the passes from the fails perfectly, there would be no bottom. A steeper curve would always lower the log-loss, and w would grow without limit. Our passes and fails overlap, so a finite best answer exists.
-
From a probability to a class
Pick a threshold t and predict a pass whenever p̂ is at least t. Moving t trades one kind of mistake for another.
To act on a probability we need a yes or no. With the fitted curve and t = 0.5, the model predicts a pass from 5.06 hours up. Rings mark the wrong predictions.
Threshold t = 0.50 Predicted fail (0) pass (1) Actual fail (0) TN 14 FP 3 pass (1) FN 3 TP 10 This table is a confusion matrix. A true positive (TP) is a predicted pass who passed, a false positive (FP) a predicted pass who failed. True negatives (TN) and false negatives (FN) are the same for predicted fails.
Precision: of the students predicted to pass, the share who passed. Recall: of the students who passed, the share predicted to pass. At t = 0.5 precision is 76.9%, recall 76.9% and accuracy 80.0%.
Move t up and fewer students are predicted to pass. Recall can only fall or stay put; precision usually rises, but not always. Try t = 0.20 and then 0.25: precision falls from 72.2% to 70.6%, because the student who drops out of the predicted passes really passed. Which t is right depends on what each kind of mistake costs. Later in the path we come back to this.
The maths
Raising can only move students from predicted pass to predicted fail. So TP cannot rise, while , the number who passed, is fixed: recall never rises. Precision usually rises, but it can dip, because a step may drop a true pass before a false one. If no one is predicted to pass, precision is 0 ÷ 0 and undefined.
With one feature and , is the same as
which is the cut-off line on the plot.
-
Two features
With two features the score is w₁x₁ + w₂x₂ + b. The boundary where p̂ = 0.5 is a straight line.
New made-up data: 80 cases with two features, 40 of each class. Circles are class 0 and triangles class 1.
The model is p̂ = σ(w·x + b), where w·x = w₁x₁ + w₂x₂. Fitting as before gives w₁ = 2.19, w₂ = 2.08 and b = −11.09.
The shading shows p̂: the stronger the violet, the closer to 1; the stronger the cyan, the closer to 0. The solid line is where p̂ = 0.5, that is w·x + b = 0. It is straight, so logistic regression is a linear classifier. The dotted lines, where p̂ = 0.1 and 0.9, run parallel to it.
With t = 0.5, accuracy on these 80 training cases is 87.5%. No straight line puts every circle on one side and every triangle on the other, so some cases are always on the wrong side.
The maths
σ here is the sigmoid, not a standard deviation. Every line has the same direction, so the lines of equal are parallel. The log-odds equal times the signed distance from the boundary, so a larger means a sharper change from 0 to 1.
If a straight line separates the two classes perfectly, there are no finite maximum-likelihood weights: scaling and up always lowers the log-loss, and the weights grow without limit. The probability map would turn into a hard step. Here no straight line puts every circle on one side and every triangle on the other, so a finite best answer exists: the fit settles, with its gradient at zero to within 10⁻⁸.
The shading is worked out on a grid of 60 × 56 small squares, each coloured by at its centre. The lines are drawn exactly.
Quick check
Lesson complete
- Logistic regression works out a straight-line score and squashes it with the sigmoid into a probability p̂ between 0 and 1.
- It is fitted by making the log-loss as small as possible. The loss is convex, so gradient descent cannot get stuck in a wrong dip.
- A threshold t turns p̂ into a class. Raising t never raises recall and usually raises precision.
- The boundary w·x + b = 0 is a straight line. If the classes can be split perfectly by a line, the weights grow without limit.
Common misconceptions
- "It is called regression, so it predicts a number like a mark." It predicts a probability, and with a threshold a class. The name comes from the logistic curve and the straight-line score inside it.
- "0.5 is the right threshold." It is a default. The right one depends on what each kind of mistake costs.
- "An S-shaped curve gives a curved boundary." The boundary is where the score is zero, and the score is a straight line.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 4.2, "Why Not Linear Regression?", and Section 4.3, "Logistic Regression", which fits the model by maximum likelihood and extends it to several features.
- The same book, Section 4.4.2, "Linear Discriminant Analysis for p > 1". Its Tables 4.6 and 4.7 name the cells of a confusion matrix and the measures built from them, including precision and recall, and the section shows what moving the threshold away from 0.5 does.
About the data. The students are made up. They are made in your browser by the mulberry32 random number generator, seed 51. For each of the 30 students in turn, the hours x are drawn uniformly between 0 and 10, then a second number u is drawn uniformly between 0 and 1, and the student passes (y = 1) if u < σ(−5 + x). So the true chance of passing is 0.5 at 5 hours. The fitted b = −5.41 and w = 1.07 are close to the −5 and 1 that made the data, but not equal: 30 students is a small sample. The two-feature data in step 6 use seed 52: feature 1 = m₁ + z₁ and feature 2 = m₂ + z₂, where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1. Class 0 has centre (m₁, m₂) = (2, 2) and class 1 has centre (4, 3.5), with 40 cases each, class 0 first.
If this helped, you can .