ML Explorer · Level 3 · Classic ML
Naive Bayes
Describe each class on its own, then let Bayes' rule say which class a new case most likely came from.
Overview
Logistic regression learns the boundary between the classes directly. Naive Bayes goes the other way round. It learns what each class looks like, feature by feature, and then asks: for a case like this one, which class is more likely?
We start with plain counts, swap them for smooth curves, slide the class shares, move to two features, and finish with data that breaks the method's main assumption.
The points are made up. See "About the data" at the end.
You'll learn
- how Bayes' rule combines a prior and a likelihood into the probability of each class
- what the "naive" assumption is and why it gives axis-aligned ovals, and why different spreads curve the boundary
- why Naive Bayes can label well yet be too sure of its probabilities
-
Bayes' rule with counts
Count how common each class is, and how common each class is near a given value. Bayes' rule turns those counts into the chance of each class.
There are 100 training cases: 60 in class 0 (cyan circles) and 40 in class 1 (violet triangles). Each has two features. For the first three steps we use only feature 1, written x₁. The small markers under the bars are the cases, one row per class.
Before we look at any feature, a case is class 1 with probability 40 ÷ 100 = 0.4. This is the prior, written P(class 1). It is estimated from the class counts, so P(class 0) = 0.6.
Now take only the cases with x₁ from 3 up to 4, inside the dashed band. There are circles and triangles, so the share of class 1 among them is ÷ = . This is the posterior, P(class 1 | x₁ in the band): the chance of class 1 once we know where the case sits. Read the bar "|" as "given".
Bayes' rule gets the same answer from three other counts. Of class 1's 40 cases, are in the band. Multiply that fraction by the prior, 0.4, and divide by the share of all cases in the band, out of 100. The result is again.
The bars count every band of width 1 in the same way. Class 0 is the solid bar on the left of each pair, class 1 the hatched bar on the right. Each bar's height is its share of all 100 cases, so the two bars in the dashed band are and tall.
The maths
Bayes' rule for a class and an observation :
With counts, let be the number of cases, the number in class , the number in the band and the number of class- cases in the band. Then
so Bayes' rule and direct counting agree exactly. The rule earns its keep when we cannot count directly: when the band is too narrow to hold any cases, or when there are many features.
-
A smooth curve for each class
Swap the bars for one smooth curve per class. Then we can read the chance of each class at any value of x₁, not just band by band.
For each class, Naive Bayes fits a normal curve, the bell shape, to that class's values of x₁, using the class's own mean and standard deviation. The fitted curve, p(x₁ | class), is the likelihood: how typical the value x₁ is of that class.
Each curve is drawn multiplied by its prior, 0.6 for class 0 (solid) and 0.4 for class 1 (dashed), so it sits over that class's bars. The area under the class 0 curve is 0.6 and under the class 1 curve 0.4. Class 1's curve is narrower and, even after multiplying by its smaller prior, taller, because its cases are less spread out.
The curve in the lower panel is the posterior, P(class 1 | x₁): the height of the class 1 curve divided by the sum of both heights. Move the x₁ slider to read both heights and the posterior at any point.
At x₁ = 3.5 the class 0 curve is higher, and P(class 1 | x₁) = . The curves cross at x₁ = , where the posterior is exactly 0.5. That is the decision boundary: to its right, Naive Bayes predicts class 1.
The maths
For class , with mean and variance of feature 1, the normal curve is
Here is a standard deviation. The fit uses the class's cases, and the variance divides by , not : it is the population variance, which is also the maximum-likelihood estimate.
The posterior is the weighted height of one curve over the sum of both:
A normal curve is a density, not a probability: its height can be above 1. The curves on the plot are densities multiplied by priors. Because the two classes have different spreads, the log of the ratio of the weighted curves is a quadratic in , so the curves can cross at two points, and here they do. The second crossing is at ≈ , off the right of the plot. Beyond it the wider class 0 curve is higher again, and Naive Bayes would predict class 0. No training case lies beyond 6.
-
The prior moves the boundary
Change the prior and the boundary slides, although neither curve changes its shape.
The slider sets P(class 1) by hand, from 0.1 to 0.9. P(class 0) is 1 minus it. The value counted from the data is 0.4.
Raise P(class 1) and the class 1 curve grows taller while the class 0 curve shrinks. Their crossing moves left, towards class 0, so more values of x₁ are predicted class 1. Lower it and the boundary moves right.
At the counted prior, 0.4, the boundary is at x₁ = . At 0.8 it is at x₁ = .
Why set a prior by hand? Because the training data may not have the class shares you will meet in use. If one class was deliberately over-sampled for training, the prior should be set to its real share.
The maths
Take logs of the ratio of the two posteriors (the log-odds):
The prior only adds a constant. Raising raises the log-odds by the same amount at every , so the posterior goes up everywhere and the stretch where it is above 0.5 can only grow. The boundary is where the log-odds is 0.
-
Two features: the naive step
With two features, Naive Bayes fits one curve per feature and multiplies them. Treating the features as unrelated within each class is the "naive" part.
Now both features count. A case is x = (x₁, x₂). For each class, fit a normal curve to feature 1 and another to feature 2, then multiply: p(x | class) = p(x₁ | class) × p(x₂ | class).
That product is right only if, within each class, knowing x₁ tells you nothing more about x₂. This is called independence, and assuming it is why the method is called naive. In this data it holds: each class was made with no link between its two features.
The ovals mark where each class's product is constant, 1 and 2 standard deviations from the class mean along each feature: solid for class 0, dashed for class 1. They line up with the axes and can never tilt, because the product has no term that links x₁ and x₂.
Class 0 has means and , with standard deviations and . Class 1 has means and , with standard deviations and .
Drag the new case, or tab to it and use the arrow keys. For each class the read-out multiplies the prior by the two curve heights. P(class 1 | x) is the class 1 product divided by the sum of both. At the starting spot, (3.5, 2.5), it is .
The maths
With features and the naive assumption,
where means "proportional to": divide by the sum over both classes to get probabilities that add to 1. Each is a normal curve with its own mean and variance , the variance again dividing by .
The product is constant where is constant. That is an ellipse with its axes along the feature axes; the ovals drawn are where the sum is 1 and 4.
A product of many small numbers can be too small for a computer to store and becomes 0. So the page adds logs instead, , and turns them back into probabilities with the log-sum-exp step, where :
Subtracting first keeps every exponent at 0 or below, so nothing overflows, and the largest term is , so the sum never becomes 0.
-
The boundary
Shade every spot by the class Naive Bayes predicts. The boundary curves, because the two classes have different spreads.
The cyan tint is predicted class 0 and the violet tint class 1. The tint is stronger where the model is surer, and a line marks the boundary, where P(class 1 | x) = 0.5.
Class 1 is narrower than class 0 in both features: standard deviations and , against and . So class 1's product falls away faster as you move from its centre, and the boundary bends round it.
On test cases from the same source, test accuracy is .
The prior slider is back. Raise P(class 1) and the violet region grows.
The maths
The log-odds is a sum over the features, and each term has a squared part:
If both classes had the same variance in each feature, every squared term would cancel and the boundary would be a straight line, of the same form as logistic regression's, . Here the variances differ, so the boundary is curved.
The shading is a simplification: it is worked out on a grid of 64 × 56 small squares, each coloured by the prediction at its centre, so the drawn boundary follows the grid.
-
When independence fails
When two features move together within a class, Naive Bayes counts the same evidence twice. Its probabilities tend to become too sure of themselves.
This is a second data set, made in the same way except that within each class feature 2 rises with feature 1. Correlation measures how closely two features rise and fall together, from −1 to 1. In the training cases it is in class 0 and in class 1. The clouds are tilted, but Naive Bayes's ovals cannot tilt.
For each test case, call the probability the model gives to the class it picks its confidence. If those probabilities are honest, the average confidence should be close to the share of cases it gets right.
Press "Independent features" for the data from steps 1–5. There the two match within a point: mean confidence , test accuracy . On the correlated data the mean confidence is , but the test accuracy is only . The model is percentage points too sure.
The reason: when x₂ moves with x₁, a high x₂ mostly repeats what a high x₁ already said. Multiplying the two curves counts that evidence twice, which pushes the probabilities towards 0 and 1.
The labels hold up better. The rule built from the true tilted shapes, the best possible on average, scores on the same test cases. Naive Bayes is points behind it, less than its confidence is off by. Trust its labels more than its probabilities.
The maths
Take the extreme case, a feature copied twice, . The product is , so the log-odds term for that feature is counted twice and the posterior is pushed further from 0.5 than the evidence justifies. With a correlation of 0.85 the double count is partial, not complete.
The true-shape rule uses the real means, standard deviations, correlation (0.85) and class shares (0.6 and 0.4) that made the data. On average over new cases no rule makes fewer mistakes, but in practice you never know those values; it is here only as a yardstick.
Mean confidence is the average over the test cases of . Comparing it with accuracy is a rough check of calibration: whether the probabilities mean what they say.
Quick check
Lesson complete
- Bayes' rule: the posterior P(class | x) is proportional to the prior P(class) times the likelihood p(x | class).
- The prior is estimated from the class counts; the likelihood comes from a curve fitted to each class. Changing the prior moves the boundary.
- The naive assumption treats the features as independent within each class, so the likelihood is a product of one curve per feature.
- When features move together, Naive Bayes counts the same evidence twice: the labels can stay good while the probabilities become too sure.
Common misconceptions
- "Naive Bayes only works when the features are independent." Its labels are often good anyway; it is its probabilities that suffer.
- "The prior is a formality." It shifts every posterior and moves the boundary.
- "A likelihood is the probability of the class." It is the other way round: p(x | class) is how typical x is of the class. P(class | x) is the posterior.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 4.4.4, "Naive Bayes", which states the independence assumption, writes the posterior as a product of one density per predictor, and explains why the method often works well even when the assumption is false.
About the data. The cases are made up. In each class, feature 1 = m₁ + s z₁ and feature 2 = m₂ + s (ρ z₁ + √(1 − ρ²) z₂), where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1, made with mulberry32. Class 0 has centre (m₁, m₂) = (2, 2) and s = 1; class 1 has centre (4.5, 3.5) and s = 0.7. Each set draws class 0 first, then class 1, from one stream. Steps 1–5 use ρ = 0: 100 training cases (60 class 0, 40 class 1) with seed 61 and 1000 test cases (600 and 400) with seed 63. Step 6 uses ρ = 0.85 in both classes: 100 training cases with seed 62 and 1000 test cases with seed 64. The new case starts at (3.5, 2.5). Each class's variances divide by its own count, nc. The up-and-down spread of the small markers in steps 1–3 (seed 5) is only so they do not hide each other. Displayed values are rounded; the calculations are not.
If this helped, you can .