Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 3 · Classic ML

Logistic regression

Squash a straight-line score into a probability, then pick a threshold to turn it into a class.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

In Lesson 11, k-NN gave each new case a label by a vote. Logistic regression gives something more useful first: a probability. "This student has a 0.8 chance of passing." A threshold then turns that probability into a label.

We go back to students and study hours, as in Lesson 1, but now we only know whether each student passed. We see why a straight line is the wrong shape, bend it into an S, score it, fit it, and then decide where to draw the line between pass and fail.

The students are made up. See "About the data" at the end.

You'll learn

  • why a straight line is the wrong shape for a probability, and how the sigmoid curve fixes it
  • how log-loss scores a probability, and how fitting finds the best curve
  • how a threshold turns a probability into a class, and what precision and recall measure
  1. Why a straight line fails

    Write a pass as 1 and a fail as 0. Fit a straight line to those 0s and 1s, and it soon predicts values no chance can take.

    Each point is one student: hours studied across, the result up. A pass is y = 1, a violet triangle. A fail is y = 0, a cyan circle. Of the 30 students, 13 passed and 17 failed.

    More hours go with more passes, but not neatly. Some students who studied little passed, and some who studied a lot failed.

    The dashed line is the least-squares straight line through the 0s and 1s, fitted as in Lesson 6. We would like to read its height as the chance of passing.

    It gives −0.19 at 0 hours and 1.17 at 10 hours. A chance cannot be below 0 or above 1. The line drops below 0 under 1.4 hours and rises above 1 past 8.7 hours.

    We need a curve that stays between 0 and 1: low for few hours, high for many, and rising in between.

    The maths

    From this lesson on we write a straight line as

    b+w xb + w\,x

    with bb the intercept and ww the slope, which machine learning calls a weight. Lesson 1 wrote a+b xa + b\,x, with bb as the slope; here bb has a new job.

    Whatever the data, a line with w≠0w \neq 0 leaves the range 0 to 1 once xx is far enough out, because b+w xb + w\,x keeps growing in one direction. On these data it already leaves inside the 0 to 10 hours we observed.

  2. The sigmoid

    Work out a straight-line score, then squash it between 0 and 1 with the sigmoid. The result is a probability.

    For a student who studied x hours, the score is z = b + w·x, a straight line. The sigmoid σ turns any score into a number between 0 and 1, so the model is p̂ = σ(b + w·x). σ here is the sigmoid, not a standard deviation.

    p̂, said "p-hat", is the predicted probability of class 1. Here that is the chance of passing.

    Move the sliders. b slides the curve left and right. w sets how steep it is: a larger w gives a sharper change from likely fail to likely pass. The middle of the curve sits at −b ÷ w hours, so changing w moves the middle too, unless b changes with it. A negative w turns the curve round, so more hours would mean a lower chance.

    The curve crosses 0.5 where the score is 0, at x = −b ÷ w hours. A large score gives p̂ close to 1 and a very negative one p̂ close to 0, but never exactly 0 or 1.

    The maths

    The sigmoid, also called the logistic function, is

    σ(z)=11+e−z,p^=σ(b+w x)\sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \hat{p} = \sigma(b + w\,x)

    σ here is the sigmoid, not a standard deviation. It has σ(0)=12\sigma(0) = \tfrac{1}{2} and σ(−z)=1−σ(z)\sigma(-z) = 1 - \sigma(z), so the curve is symmetric about its middle.

    Turn it inside out. The odds p^/(1−p^)\hat{p}/(1-\hat{p}) compare the chance of passing with the chance of failing, and their natural log, the log-odds, is the straight line:

    ln⁡p^1−p^=b+w x\ln\frac{\hat{p}}{1 - \hat{p}} = b + w\,x

    So each extra hour adds ww to the log-odds, which multiplies the odds by ewe^{w}. The model is linear in the log-odds, not in the probability.

  3. Log-loss: scoring a probability

    Score each student by the probability the curve gave to what really happened. A confident wrong answer costs the most.

    The bars under the plot are each student's penalty. For a student who passed, it is −ln p̂. For one who failed, it is −ln(1 − p̂). ln is the natural logarithm.

    If the curve gave what happened a probability near 1, the penalty is near 0. A probability of 0.5 costs 0.69, and 0.01 costs 4.6. As the probability goes to 0, the penalty grows without limit.

    The mean of the penalties is the log-loss. Move the sliders and watch it change. The ringed bar is the largest penalty.

    At the best curve, found in the next step, the largest penalty is 1.56. It belongs to the student who studied 6.30 hours and failed: the curve gave them a 0.79 chance of passing.

    The maths

    The mean log-loss over nn students is

    L(b,w)=−1n∑i=1n[ yiln⁡p^i+(1−yi)ln⁡(1−p^i)]L(b, w) = -\frac{1}{n}\sum_{i=1}^{n} \Big[\, y_i \ln \hat{p}_i + (1 - y_i)\ln(1 - \hat{p}_i) \Big]

    For each student one of the two terms is zero, since yiy_i is 0 or 1. With b=w=0b = w = 0 every p^i=0.5\hat{p}_i = 0.5 and the log-loss is ln⁡2\ln 2, about 0.693.

    Making the log-loss small is the same as making the product of the probabilities given to what happened large, because e−nLe^{-nL} is that product. Choosing the model this way is called maximum likelihood.

    Why not the squared miss of Lesson 4? On top of the sigmoid it is not convex in bb and ww in general, and it never charges more than 1 for a wrong answer, however confident. Log-loss is convex, and it punishes confident mistakes hard.

    The page clips p^\hat{p} to the range [10−15, 1−10−15][10^{-15},\, 1 - 10^{-15}] before taking the log, so every penalty stays finite. Bars taller than 3 are cut and marked with two short lines.

  4. Fitting: walking downhill

    Training finds the b and w with the smallest log-loss. Gradient descent gets there one small step at a time.

    Press Fit. Gradient descent starts at b = 0 and w = 0, where the curve is flat at 0.5 and the log-loss is 0.693. Each step moves b and w a little way downhill.

    The chart shows the log-loss after each step. It falls fast at first, then slowly. With learning rate η = 0.25 it takes 767 steps to come within 0.001 of the minimum, and after 3000 steps it is within 0.000001.

    The dashed curve is the answer from Newton's method, a faster method that also uses how the slope of the loss changes. It needs 6 steps: b = −5.41, w = 1.07, log-loss 0.328. That curve crosses 0.5 at 5.06 hours.

    There is no trap to fall into. The log-loss is convex: one bowl with no other dips, so walking downhill from any start ends at the same bottom.

    The maths

    The gradient of the log-loss has a simple form: the miss p^i−yi\hat{p}_i - y_i times the input.

    ∂L∂b=1n∑i=1n(p^i−yi),∂L∂w=1n∑i=1n(p^i−yi) xi\frac{\partial L}{\partial b} = \frac{1}{n}\sum_{i=1}^{n} (\hat{p}_i - y_i), \qquad \frac{\partial L}{\partial w} = \frac{1}{n}\sum_{i=1}^{n} (\hat{p}_i - y_i)\,x_i

    Each step is b←b−η ∂L/∂bb \leftarrow b - \eta\,\partial L/\partial b and w←w−η ∂L/∂ww \leftarrow w - \eta\,\partial L/\partial w. Unlike least squares, setting the gradient to zero has no formula for the answer, so it is found step by step.

    The second derivatives form the matrix

    1n∑i=1np^i(1−p^i)(1xixixi2)\frac{1}{n}\sum_{i=1}^{n} \hat{p}_i(1 - \hat{p}_i)\begin{pmatrix} 1 & x_i \\ x_i & x_i^2 \end{pmatrix}

    which is never negative in any direction, so the loss is convex. Newton's method multiplies the gradient by the inverse of this matrix (with one weight it would divide by the curvature); gradient descent uses only the gradient. Gradient descent is slow here because the hours are far from 0: they are not centred, so a change in w also moves the curve's middle, and b has to follow. That ties b and w together and gives the long, narrow valley of Lesson 8. With η = 0.25 every step lowers the loss on these data.

    If some number of hours split the passes from the fails perfectly, there would be no bottom. A steeper curve would always lower the log-loss, and w would grow without limit. Our passes and fails overlap, so a finite best answer exists.

  5. From a probability to a class

    Pick a threshold t and predict a pass whenever p̂ is at least t. Moving t trades one kind of mistake for another.

    To act on a probability we need a yes or no. With the fitted curve and t = 0.5, the model predicts a pass from 5.06 hours up. Rings mark the wrong predictions.

    Threshold t = 0.50
    Predicted
    fail (0)pass (1)
    Actualfail (0)TN 14FP 3
    pass (1)FN 3TP 10

    This table is a confusion matrix. A true positive (TP) is a predicted pass who passed, a false positive (FP) a predicted pass who failed. True negatives (TN) and false negatives (FN) are the same for predicted fails.

    Precision: of the students predicted to pass, the share who passed. Recall: of the students who passed, the share predicted to pass. At t = 0.5 precision is 76.9%, recall 76.9% and accuracy 80.0%.

    Move t up and fewer students are predicted to pass. Recall can only fall or stay put; precision usually rises, but not always. Try t = 0.20 and then 0.25: precision falls from 72.2% to 70.6%, because the student who drops out of the predicted passes really passed. Which t is right depends on what each kind of mistake costs. Later in the path we come back to this.

    The maths
    y^={1if p^≥t0otherwiseprecision=TPTP+FP,recall=TPTP+FN\hat{y} = \begin{cases} 1 & \text{if } \hat{p} \ge t \\ 0 & \text{otherwise} \end{cases} \qquad \text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}

    Raising tt can only move students from predicted pass to predicted fail. So TP cannot rise, while TP+FNTP + FN, the number who passed, is fixed: recall never rises. Precision usually rises, but it can dip, because a step may drop a true pass before a false one. If no one is predicted to pass, precision is 0 ÷ 0 and undefined.

    With one feature and w>0w > 0, p^≥t\hat{p} \ge t is the same as

    x≥ln⁡(t/(1−t))−bwx \ge \frac{\ln\big(t/(1-t)\big) - b}{w}

    which is the cut-off line on the plot.

  6. Two features

    With two features the score is w₁x₁ + w₂x₂ + b. The boundary where p̂ = 0.5 is a straight line.

    New made-up data: 80 cases with two features, 40 of each class. Circles are class 0 and triangles class 1.

    The model is p̂ = σ(w·x + b), where w·x = w₁x₁ + w₂x₂. Fitting as before gives w₁ = 2.19, w₂ = 2.08 and b = −11.09.

    The shading shows p̂: the stronger the violet, the closer to 1; the stronger the cyan, the closer to 0. The solid line is where p̂ = 0.5, that is w·x + b = 0. It is straight, so logistic regression is a linear classifier. The dotted lines, where p̂ = 0.1 and 0.9, run parallel to it.

    With t = 0.5, accuracy on these 80 training cases is 87.5%. No straight line puts every circle on one side and every triangle on the other, so some cases are always on the wrong side.

    The maths
    p^=σ(w⋅x+b),ln⁡p^1−p^=w⋅x+b\hat{p} = \sigma(\mathbf{w}\cdot\mathbf{x} + b), \qquad \ln\frac{\hat{p}}{1-\hat{p}} = \mathbf{w}\cdot\mathbf{x} + b

    σ here is the sigmoid, not a standard deviation. Every line w⋅x+b=c\mathbf{w}\cdot\mathbf{x} + b = c has the same direction, so the lines of equal p^\hat{p} are parallel. The log-odds equal ∥w∥\lVert \mathbf{w} \rVert times the signed distance from the boundary, so a larger ∥w∥\lVert \mathbf{w} \rVert means a sharper change from 0 to 1.

    If a straight line separates the two classes perfectly, there are no finite maximum-likelihood weights: scaling w\mathbf{w} and bb up always lowers the log-loss, and the weights grow without limit. The probability map would turn into a hard step. Here no straight line puts every circle on one side and every triangle on the other, so a finite best answer exists: the fit settles, with its gradient at zero to within 10⁻⁸.

    The shading is worked out on a grid of 60 × 56 small squares, each coloured by p^\hat{p} at its centre. The lines are drawn exactly.

Quick check

For one student the model gives p̂ = 0.8. What does that mean?

Right. p̂ is the model's estimated probability of class 1 for this student. It is only as good as the model: if its probabilities can be trusted, then of many students given 0.8, about 8 in 10 pass.

p̂ belongs to one student, worked out from their hours. The share of all students who passed is a single number for the whole group.

That would be accuracy, a score for the model over many cases. p̂ is a probability for one case.

A probability of 0.8 still leaves a 0.2 chance of failing. Whether the model predicts a pass depends on the threshold: with t = 0.9 it would predict a fail.

On this lesson's students, what does raising the threshold from 0.5 to 0.8 do to precision and recall?

Recall cannot rise when the threshold goes up: fewer students are predicted to pass, so no extra passes are caught.

Right. Precision goes from 76.9% to 100% and recall from 76.9% to 53.8%. At t = 0.8 only the most confident are predicted to pass: all 7 of 7 did, but they are only 7 of 13 who passed.

That is what lowering the threshold tends to do. Raising it predicts fewer passes, so recall falls.

The curve is the same, but the threshold decides which students are predicted to pass, and that changes the counts.

Lesson complete

  • Logistic regression works out a straight-line score and squashes it with the sigmoid into a probability p̂ between 0 and 1.
  • It is fitted by making the log-loss as small as possible. The loss is convex, so gradient descent cannot get stuck in a wrong dip.
  • A threshold t turns p̂ into a class. Raising t never raises recall and usually raises precision.
  • The boundary w·x + b = 0 is a straight line. If the classes can be split perfectly by a line, the weights grow without limit.

Common misconceptions

  • "It is called regression, so it predicts a number like a mark." It predicts a probability, and with a threshold a class. The name comes from the logistic curve and the straight-line score inside it.
  • "0.5 is the right threshold." It is a default. The right one depends on what each kind of mistake costs.
  • "An S-shaped curve gives a curved boundary." The boundary is where the score is zero, and the score is a straight line.
Next lesson: Naive Bayes →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 4.2, "Why Not Linear Regression?", and Section 4.3, "Logistic Regression", which fits the model by maximum likelihood and extends it to several features.
  2. The same book, Section 4.4.2, "Linear Discriminant Analysis for p > 1". Its Tables 4.6 and 4.7 name the cells of a confusion matrix and the measures built from them, including precision and recall, and the section shows what moving the threshold away from 0.5 does.

About the data. The students are made up. They are made in your browser by the mulberry32 random number generator, seed 51. For each of the 30 students in turn, the hours x are drawn uniformly between 0 and 10, then a second number u is drawn uniformly between 0 and 1, and the student passes (y = 1) if u < σ(−5 + x). So the true chance of passing is 0.5 at 5 hours. The fitted b = −5.41 and w = 1.07 are close to the −5 and 1 that made the data, but not equal: 30 students is a small sample. The two-feature data in step 6 use seed 52: feature 1 = m₁ + z₁ and feature 2 = m₂ + z₂, where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1. Class 0 has centre (m₁, m₂) = (2, 2) and class 1 has centre (4, 3.5), with 40 cases each, class 0 first.

If this helped, you can .