ML Explorer · Level 5 · Modern AI
Generative models
A generative model learns where the data falls, as a probability distribution, and then makes new data by sampling from it. Fit one live with EM and press Sample.
Overview
The last lesson, Large language models, in miniature, made new text by sampling one character at a time from learned probabilities. That is one kind of generative model. This lesson shows the idea on points, where every step can be drawn: describe the data with a probability distribution, fit it by maximum likelihood, and sample from it.
The model is a mixture of Gaussians, fitted with the EM algorithm. It is small enough to watch, and it shows the questions every generative model faces: how good is the fit, how many parts does it need, and what can it not describe?
You'll learn
- what a generative model is, and how fitting one means maximising the likelihood of the data
- how EM fits a mixture of Gaussians, and why held-out data chooses the number of components
- how to sample new points, and where a few Gaussians are not enough
-
Making more data
A classifier predicts a label for a point. A generative model describes where the points fall, so it can make more of them.
On the left are 300 training points. On the right are 300 new points, drawn from a model fitted to the left-hand points in step 4. None of them is in the data: they were drawn from the model, not picked from the data.
A classifier, such as logistic regression or the networks of Level 4, is given a point and predicts its label. It says nothing about which points are likely in the first place. A generative model learns that: a probability distribution p(x) over the points themselves.
The maths
A classifier models : the label given the point. A generative model learns , or when there are labels. Naive Bayes was in fact generative: it modelled for each class and used Bayes' rule to classify. With you can sample new points, and score how likely any point is.
-
One Gaussian, by maximum likelihood
The simplest generative model is one bell curve. Fitting it means choosing the curve under which the data is most likely.
Here are 300 values on a line, with two humps. The curve is the Gaussian that makes these values most likely: its mean is the mean of the values, 0.36, and its variance is their population variance (dividing by n), 2.39.
It misses the shape. The curve is highest at the mean, between the humps, where there are few values. Press Sample to draw 300 values from it: they make one hump, not two, because one Gaussian can only make one.
To score the fit, take 300 new values from the same rule, never used for fitting (held-out values), and average the log of the density the model gives each one. Higher is better. The Gaussian scores −1.91 per value. The two-hump rule that made the data scores −1.64 on the same values, 0.27 higher.
The maths
The likelihood of parameters is the density of the data under them, the values taken as independent: . Its log, the log-likelihood , is easier to work with and has the same maximum. For one Gaussian the maximum is at
the mean and the population variance (n, not n − 1). The scores on this page are the mean log-likelihood per point, , with natural logs. A density can be above 1, so a log-likelihood can have either sign; only compare scores on the same points.
-
A mixture of Gaussians
Several Gaussians, each with a weight, can describe data with several groups.
Now the points are in two dimensions: the 300 training points from step 1, made from three Gaussians, a third of the points each. One Gaussian fitted to them (solid ellipse) covers all three groups and the empty space between them. Its mean log-likelihood on these points is −3.93.
A mixture adds K Gaussians together. Each component k has a mean μk, a covariance Σk that sets its ellipse's size, shape and tilt, and a weight πk (a weight here, not 3.14…): the share of points it accounts for. The dashed ellipses are the three Gaussians that made the data, weights 1/3; as a mixture they score −3.08.
Each ellipse marks the points at Mahalanobis distance 2 from its mean. It holds about 86% of the points that Gaussian makes.
The maths
To sample one point: pick component k with probability , then draw from its Gaussian. The ellipse is the set of with ; in two dimensions a Gaussian puts of its probability inside it.
-
Fitting with EM
Press Train to run EM with K = 3: it shares each point among the components, then refits each component to its share, again and again.
EM starts with three means at training points picked at random, every covariance equal to the whole data's (the sample covariance, dividing by n − 1) and every weight 1/3. Each iteration has two steps. The E-step works out, for every point, the probability that each component made it: its responsibilities, which add up to 1. The M-step refits each Gaussian by maximum likelihood, every point counted with its responsibility as a weight.
Each point is drawn as the shape of the component with its largest responsibility, and faded when it is shared. The curve is the mean log-likelihood after each iteration, from −4.37 at the start. It rises fast, nearly stops for a few iterations, then climbs again as the ellipses separate the groups, and it never falls. The run stops when one iteration adds less than 0.000001: here after 29 iterations, at −3.05.
EM is like k-means with soft assignments: k-means gives each point wholly to its nearest centre; EM shares it out by probability, and also learns each group's shape and weight. Like k-means, EM can stop at a local maximum: a different start can end in a different, worse fit.
The maths
E-step, the responsibility of component k for point i:
M-step, with :
No iteration can lower the log-likelihood ; Dempster, Laird and Rubin (1977) proved this for EM in general. It does not promise the highest possible value, only a value that stops rising. Nothing is added to the covariances here; if a component shrank onto a single point its covariance would become singular, which does not happen in any run on this page.
-
How many components?
Slide K from 1 to 5. Each K is fitted with EM on the training points and scored on held-out points.
Every K starts the same way as in step 4 and uses the same stopping rule. On the training points the score rises with every extra component here: −3.93, −3.38, −3.05, −3.03, −3.01 for K = 1 to 5. More components can always bend closer to the training points.
The held-out points are 300 new points from the same rule. Their scores are −3.92, −3.43, −3.17, −3.19 and −3.19: best at K = 3, the number of Gaussians that made the data, though K = 4 and 5 are only 0.02 lower. Another set of held-out points can put K = 4 first. The extra components of K = 4 and 5 fit quirks of the training points that new points do not share, as in overfitting. For comparison, the rule itself scores −3.16 on the held-out points.
Here the groups are well apart and held-out data picks the true K, but only just. That is not a rule: with groups that overlap, several K often score almost the same.
The maths
A mixture of K Gaussians in two dimensions has 6K − 1 numbers to fit: 2 for each mean, 3 for each covariance, and K weights that add up to 1. A bigger mixture can always copy a smaller one, for example by splitting one component into two identical halves of its weight, so at their best fit more components can always score at least as well on the training points. The training score can even grow without limit if one component shrinks onto a single point. So the training score cannot choose K; held-out data, or a penalty for the number of parameters, can.
-
Sampling, and its limit
Press Sample for new points from each fitted mixture. On three blobs they look right; on a spiral, five Gaussians make a blurred disc, not a spiral.
Left: new points from the K = 3 fit of step 4. Right: a spiral of 400 points, fitted with K = 5 in the same way, and 400 new points from it. Two numbers judge the new points. How far is each new point from the nearest data point? Small means they land where the data is. How far is each data point from the nearest new point? Small means they cover all of the data. Call the first precision and the second coverage: the names say what each checks, though here smaller is better for both.
For the blobs, over the first 20 draws the new points sit 0.14 to 0.16 from the nearest data point, and the data points 0.13 to 0.15 from the nearest new point: close to the 0.15 and 0.16 of the held-out points, made by the true rule.
For the spiral, the new points sit 0.15 to 0.18 from the data, about three times the 0.05 of fresh points from the spiral's own rule, and the data points sit 0.09 to 0.11 from the new points, against 0.06. The five ellipses are broad and overlap, so together they spread new points over the whole disc, in the empty space between the arms as well as on them.
A mixture with enough components could fit the spiral too, with many small Gaussians laid along the arms. The question is how many it needs, each with its own mean, covariance and weight to fit. Real generative models do the same job, learning to make new data like the training data, at far larger scale and with neural networks instead of a few Gaussians. The next lesson builds one that learns the spiral's shape a different way.
The maths
With samples and data , the two numbers are
Each depends on the size of the other set: the first shrinks as the data set grows, the second as more points are drawn. So compare them only at equal sizes, as here: every comparison on this page uses 300 new points against 300 data points for the blobs, and 400 against 400 for the spiral. Each press draws with the next seed.
Quick check
Lesson complete
- A generative model learns a probability distribution for the data, p(x), and makes new data by sampling from it.
- Fitting means maximising the likelihood. For a mixture of Gaussians, EM does it by alternating responsibilities (E-step) and weighted refits (M-step); the likelihood never falls, but it can stop at a local maximum.
- Choose the number of components on held-out data. A few Gaussians describe a few blobs well, but a curved shape needs many.
Common misconceptions
- "A generative model copies the training points." A few Gaussians cannot copy: their samples fall near the data, not on it. A model with one tiny component per point could copy, which is overfitting again. Either way, samples hold nothing the model did not already learn.
- "More components are always better." The training score rises, but here the held-out score is best at K = 3.
- "EM finds the best fit." It finds a fit that further iterations barely improve; another start can end somewhere worse.
Further reading
- Bishop, Pattern Recognition and Machine Learning: mixture models and EM.
- Dempster, Laird & Rubin, "Maximum Likelihood from Incomplete Data via the EM Algorithm", Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1977.
About the data. All the data is made in your browser by the mulberry32 random number generator, with Gaussian draws by the Box–Muller method. The 1-D values: for each of 300, one uniform draw u; if u < 0.4 the value is drawn from N(−1.5, 0.5²), otherwise from N(1.5, 0.75²); seed 301 for the training values and 306 for the held-out values. The 2-D blobs: point i is drawn from Gaussian i mod 3, with means (−2, −1), (2, −1) and (0, 2.2) and covariances [[0.5, 0.2], [0.2, 0.4]], [[0.4, −0.15], [−0.15, 0.6]] and [[0.7, 0], [0, 0.25]]; 300 training points from seed 302 and 300 held-out points from seed 303. EM starts from seed 304 for every fit. The spiral: 400 points from seed 305, 200 on each of two arms; along an arm the radius grows from 0.3 towards 3 over one turn, the second arm starts half a turn round from the first, and Gaussian noise with standard deviation 0.1 is added to each coordinate; each axis is then standardised to mean 0 and standard deviation 1 (dividing by n); the fresh spiral is made the same way from seed 307. Samples start at seed 308 for the 1-D values, 309 for the blobs (also the right-hand panel of step 1) and 310 for the spiral, and each press of Sample adds 1 to the seed.
If this helped, you can .