Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 3 · Classic ML

Naive Bayes

Describe each class on its own, then let Bayes' rule say which class a new case most likely came from.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

Logistic regression learns the boundary between the classes directly. Naive Bayes goes the other way round. It learns what each class looks like, feature by feature, and then asks: for a case like this one, which class is more likely?

We start with plain counts, swap them for smooth curves, slide the class shares, move to two features, and finish with data that breaks the method's main assumption.

The points are made up. See "About the data" at the end.

You'll learn

  • how Bayes' rule combines a prior and a likelihood into the probability of each class
  • what the "naive" assumption is and why it gives axis-aligned ovals, and why different spreads curve the boundary
  • why Naive Bayes can label well yet be too sure of its probabilities
  1. Bayes' rule with counts

    Count how common each class is, and how common each class is near a given value. Bayes' rule turns those counts into the chance of each class.

    There are 100 training cases: 60 in class 0 (cyan circles) and 40 in class 1 (violet triangles). Each has two features. For the first three steps we use only feature 1, written x₁. The small markers under the bars are the cases, one row per class.

    Before we look at any feature, a case is class 1 with probability 40 ÷ 100 = 0.4. This is the prior, written P(class 1). It is estimated from the class counts, so P(class 0) = 0.6.

    The maths

    Bayes' rule for a class cc and an observation xx:

    P(c∣x)=P(x∣c) P(c)P(x),P(x)=∑c′P(x∣c′) P(c′)P(c \mid x) = \frac{P(x \mid c)\,P(c)}{P(x)}, \qquad P(x) = \sum_{c'} P(x \mid c')\,P(c')

    With counts, let nn be the number of cases, ncn_c the number in class cc, nBn_B the number in the band and nc,Bn_{c,B} the number of class-cc cases in the band. Then

    P(c∣B)=(nc,B/nc) (nc/n)nB/n=nc,BnBP(c \mid B) = \frac{(n_{c,B}/n_c)\,(n_c/n)}{n_B/n} = \frac{n_{c,B}}{n_B}

    so Bayes' rule and direct counting agree exactly. The rule earns its keep when we cannot count directly: when the band is too narrow to hold any cases, or when there are many features.

  2. A smooth curve for each class

    Swap the bars for one smooth curve per class. Then we can read the chance of each class at any value of x₁, not just band by band.

    For each class, Naive Bayes fits a normal curve, the bell shape, to that class's values of x₁, using the class's own mean and standard deviation. The fitted curve, p(x₁ | class), is the likelihood: how typical the value x₁ is of that class.

    Each curve is drawn multiplied by its prior, 0.6 for class 0 (solid) and 0.4 for class 1 (dashed), so it sits over that class's bars. The area under the class 0 curve is 0.6 and under the class 1 curve 0.4. Class 1's curve is narrower and, even after multiplying by its smaller prior, taller, because its cases are less spread out.

    The curve in the lower panel is the posterior, P(class 1 | x₁): the height of the class 1 curve divided by the sum of both heights. Move the x₁ slider to read both heights and the posterior at any point.

    The maths

    For class cc, with mean μc\mu_c and variance σc2\sigma_c^2 of feature 1, the normal curve is

    p(x1∣c)=12πσc2exp⁡ ⁣(−(x1−μc)22σc2)p(x_1 \mid c) = \frac{1}{\sqrt{2\pi\sigma_c^2}}\exp\!\left(-\frac{(x_1 - \mu_c)^2}{2\sigma_c^2}\right)

    Here σ\sigma is a standard deviation. The fit uses the class's ncn_c cases, and the variance divides by ncn_c, not nc−1n_c - 1: it is the population variance, which is also the maximum-likelihood estimate.

    μc=1nc∑i: yi=cxi1,σc2=1nc∑i: yi=c(xi1−μc)2\mu_c = \frac{1}{n_c}\sum_{i:\,y_i = c} x_{i1}, \qquad \sigma_c^2 = \frac{1}{n_c}\sum_{i:\,y_i = c} (x_{i1} - \mu_c)^2

    The posterior is the weighted height of one curve over the sum of both:

    P(1∣x1)=P(1) p(x1∣1)P(0) p(x1∣0)+P(1) p(x1∣1)P(1 \mid x_1) = \frac{P(1)\,p(x_1 \mid 1)}{P(0)\,p(x_1 \mid 0) + P(1)\,p(x_1 \mid 1)}
  3. The prior moves the boundary

    Change the prior and the boundary slides, although neither curve changes its shape.

    The slider sets P(class 1) by hand, from 0.1 to 0.9. P(class 0) is 1 minus it. The value counted from the data is 0.4.

    Raise P(class 1) and the class 1 curve grows taller while the class 0 curve shrinks. Their crossing moves left, towards class 0, so more values of x₁ are predicted class 1. Lower it and the boundary moves right.

    Why set a prior by hand? Because the training data may not have the class shares you will meet in use. If one class was deliberately over-sampled for training, the prior should be set to its real share.

    The maths

    Take logs of the ratio of the two posteriors (the log-odds):

    log⁡P(1∣x1)P(0∣x1)=log⁡P(1)P(0)+log⁡p(x1∣1)p(x1∣0)\log\frac{P(1 \mid x_1)}{P(0 \mid x_1)} = \log\frac{P(1)}{P(0)} + \log\frac{p(x_1 \mid 1)}{p(x_1 \mid 0)}

    The prior only adds a constant. Raising P(1)P(1) raises the log-odds by the same amount at every x1x_1, so the posterior goes up everywhere and the stretch where it is above 0.5 can only grow. The boundary is where the log-odds is 0.

  4. Two features: the naive step

    With two features, Naive Bayes fits one curve per feature and multiplies them. Treating the features as unrelated within each class is the "naive" part.

    Now both features count. A case is x = (x₁, x₂). For each class, fit a normal curve to feature 1 and another to feature 2, then multiply: p(x | class) = p(x₁ | class) × p(x₂ | class).

    That product is right only if, within each class, knowing x₁ tells you nothing more about x₂. This is called independence, and assuming it is why the method is called naive. In this data it holds: each class was made with no link between its two features.

    The ovals mark where each class's product is constant, 1 and 2 standard deviations from the class mean along each feature: solid for class 0, dashed for class 1. They line up with the axes and can never tilt, because the product has no term that links x₁ and x₂.

    The maths

    With dd features and the naive assumption,

    P(c∣x)∝P(c)∏j=1dp(xj∣c)P(c \mid \mathbf{x}) \propto P(c)\prod_{j=1}^{d} p(x_j \mid c)

    where ∝\propto means "proportional to": divide by the sum over both classes to get probabilities that add to 1. Each p(xj∣c)p(x_j \mid c) is a normal curve with its own mean μcj\mu_{cj} and variance σcj2\sigma_{cj}^2, the variance again dividing by ncn_c.

    The product is constant where ∑j(xj−μcj)2/σcj2\sum_j (x_j - \mu_{cj})^2/\sigma_{cj}^2 is constant. That is an ellipse with its axes along the feature axes; the ovals drawn are where the sum is 1 and 4.

    A product of many small numbers can be too small for a computer to store and becomes 0. So the page adds logs instead, sc=log⁡P(c)+∑jlog⁡p(xj∣c)s_c = \log P(c) + \sum_j \log p(x_j \mid c), and turns them back into probabilities with the log-sum-exp step, where m=max⁡cscm = \max_c s_c:

    P(c∣x)=exp⁡ ⁣(sc−m−log⁡∑c′e sc′−m)P(c \mid \mathbf{x}) = \exp\!\Big(s_c - m - \log \sum_{c'} e^{\,s_{c'} - m}\Big)

    Subtracting mm first keeps every exponent at 0 or below, so nothing overflows, and the largest term is e0=1e^0 = 1, so the sum never becomes 0.

  5. The boundary

    Shade every spot by the class Naive Bayes predicts. The boundary curves, because the two classes have different spreads.

    The cyan tint is predicted class 0 and the violet tint class 1. The tint is stronger where the model is surer, and a line marks the boundary, where P(class 1 | x) = 0.5.

    The prior slider is back. Raise P(class 1) and the violet region grows.

    The maths

    The log-odds is a sum over the features, and each term has a squared part:

    log⁡P(1∣x)P(0∣x)=const+∑j(12σ0j2−12σ1j2)xj2+∑j(μ1jσ1j2−μ0jσ0j2)xj\log\frac{P(1 \mid \mathbf{x})}{P(0 \mid \mathbf{x})} = \text{const} + \sum_j \left(\frac{1}{2\sigma_{0j}^2} - \frac{1}{2\sigma_{1j}^2}\right) x_j^2 + \sum_j \left(\frac{\mu_{1j}}{\sigma_{1j}^2} - \frac{\mu_{0j}}{\sigma_{0j}^2}\right) x_j

    If both classes had the same variance in each feature, every squared term would cancel and the boundary would be a straight line, of the same form as logistic regression's, w⋅x+b=0\mathbf{w}\cdot\mathbf{x} + b = 0. Here the variances differ, so the boundary is curved.

    The shading is a simplification: it is worked out on a grid of 64 × 56 small squares, each coloured by the prediction at its centre, so the drawn boundary follows the grid.

  6. When independence fails

    When two features move together within a class, Naive Bayes counts the same evidence twice. Its probabilities tend to become too sure of themselves.

    For each test case, call the probability the model gives to the class it picks its confidence. If those probabilities are honest, the average confidence should be close to the share of cases it gets right.

    The reason: when x₂ moves with x₁, a high x₂ mostly repeats what a high x₁ already said. Multiplying the two curves counts that evidence twice, which pushes the probabilities towards 0 and 1.

    The maths

    Take the extreme case, a feature copied twice, x2=x1x_2 = x_1. The product is p(x1∣c)2p(x_1 \mid c)^2, so the log-odds term for that feature is counted twice and the posterior is pushed further from 0.5 than the evidence justifies. With a correlation of 0.85 the double count is partial, not complete.

    The true-shape rule uses the real means, standard deviations, correlation (0.85) and class shares (0.6 and 0.4) that made the data. On average over new cases no rule makes fewer mistakes, but in practice you never know those values; it is here only as a yardstick.

    Mean confidence is the average over the test cases of max⁡cP(c∣x)\max_c P(c \mid \mathbf{x}). Comparing it with accuracy is a rough check of calibration: whether the probabilities mean what they say.

Quick check

What does "naive" assume?

Right. That is what lets the method multiply one curve per feature. It is the part that fails on correlated data, where the probabilities came out too sure.

If the features told you nothing about the class, there would be nothing to learn. The assumption is about features among themselves, once the class is known.

No. The prior is counted from the data, 0.6 and 0.4 here, and the boundary moves when it changes.

That is the choice of curve in this lesson's version, Gaussian Naive Bayes. Other versions use other curves, or counts, and are just as naive. The naive part is the independence.

On feature 1 alone, doubling class 1's prior (0.4 to 0.8) moves the boundary which way?

Right. It moves from x₁ = to x₁ = . P(class 0) falls to 0.2 at the same time, so the class 1 curve grows and the class 0 curve shrinks.

The other way. A larger prior makes class 1 more likely everywhere, so more values of x₁ end up on its side.

The boundary is where the posterior is 0.5, and the prior changes the posterior at every point. So the boundary moves.

There is no simple rule like that. A larger prior adds the same amount to the log-odds everywhere, which moves the boundary left, towards class 0. How far depends on how steeply the two curves change near the crossing.

Lesson complete

  • Bayes' rule: the posterior P(class | x) is proportional to the prior P(class) times the likelihood p(x | class).
  • The prior is estimated from the class counts; the likelihood comes from a curve fitted to each class. Changing the prior moves the boundary.
  • The naive assumption treats the features as independent within each class, so the likelihood is a product of one curve per feature.
  • When features move together, Naive Bayes counts the same evidence twice: the labels can stay good while the probabilities become too sure.

Common misconceptions

  • "Naive Bayes only works when the features are independent." Its labels are often good anyway; it is its probabilities that suffer.
  • "The prior is a formality." It shifts every posterior and moves the boundary.
  • "A likelihood is the probability of the class." It is the other way round: p(x | class) is how typical x is of the class. P(class | x) is the posterior.
Next lesson: Support vector machines →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 4.4.4, "Naive Bayes", which states the independence assumption, writes the posterior as a product of one density per predictor, and explains why the method often works well even when the assumption is false.

About the data. The cases are made up. In each class, feature 1 = m₁ + s z₁ and feature 2 = m₂ + s (ρ z₁ + √(1 − ρ²) z₂), where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1, made with mulberry32. Class 0 has centre (m₁, m₂) = (2, 2) and s = 1; class 1 has centre (4.5, 3.5) and s = 0.7. Each set draws class 0 first, then class 1, from one stream. Steps 1–5 use ρ = 0: 100 training cases (60 class 0, 40 class 1) with seed 61 and 1000 test cases (600 and 400) with seed 63. Step 6 uses ρ = 0.85 in both classes: 100 training cases with seed 62 and 1000 test cases with seed 64. The new case starts at (3.5, 2.5). Each class's variances divide by its own count, nc. The up-and-down spread of the small markers in steps 1–3 (seed 5) is only so they do not hide each other. Displayed values are rounded; the calculations are not.

If this helped, you can .