ML Explorer · Level 5 · Modern AI
Diffusion: from noise to data
Add noise to a spiral, step by step, until almost nothing of it is left. Train a small network to guess that noise. Then start from pure noise, take the guesses back out step by step, and watch 400 points walk back into a spiral.
Overview
In generative models, a mixture of five Gaussians was fitted to a spiral, and the new points it made filled a blurred disc instead of two arms. This lesson makes new points for the same spiral another way, called diffusion.
The idea has three parts. First, a fixed recipe adds a little noise many times, until the spiral is nearly pure noise. Second, a network is trained to look at a noisy point and guess the noise in it. Third, to make new points, start from pure noise and use the network to take the noise back out, one small step at a time.
You'll learn
- how adding a little Gaussian noise many times turns data into nearly pure noise, and how to jump straight to any step
- that the network is trained to predict the added noise, with the squared error as its loss
- how new points are made by starting from noise and removing the predicted noise step by step
-
Add noise, a little at a time
Move the slider from t = 0 to t = 50 and watch the spiral dissolve into a round cloud.
The data are the 400 points of the spiral from generative models, with each axis scaled to mean 0 and standard deviation 1. One step of noise does two things to a point: it shrinks the point a little towards the origin, and it adds a small Gaussian random nudge. How much is set by a number , the noise schedule, which grows evenly from 0.001 at step 1 to 0.2 at step 50: tiny nudges at first, bigger ones later.
On the right, the solid curve is how much of the original point is left after t steps, and the dashed curve is the standard deviation of all the noise added so far. By t = 50 the original point is multiplied by only 0.067: the spiral is gone, and what remains looks like draws from a standard Gaussian, N(0, I). Nothing here is learned. It is a fixed recipe, the same for every point.
The picture uses a shortcut, explained in the next step, which gives the same spread at every t as adding the noise one step at a time. Each point keeps one noise draw, so you can follow it as t grows.
The maths
Write for a data point and for it after t steps, with and (here T is the number of steps, not a transpose). With , one step is
that is, with a fresh draw at every step. Here . The factor keeps the spread from growing: if an axis has variance 1 before a step, it has after it, and the data start with variance 1 on each axis (population variance, dividing by n).
-
Jump straight to step t
You do not have to add the noise 25 times to see step 25. One formula gives it in one jump.
Each small step shrinks a point and adds independent Gaussian noise. Shrinking a Gaussian and adding another independent Gaussian gives a Gaussian again, so after t steps the point is a shrunk copy of the original plus one Gaussian nudge. The shrink factor is the solid curve, , and the nudge's standard deviation is the dashed one, .
For three points, the picture draws an arrow from the data point to its shrunk centre, and circles of one and two standard deviations around that centre: at step t, the noisy point lands somewhere around there. Move the slider and the centres slide towards the origin while the circles grow.
This jump is what makes training cheap. Training needs many noisy points at many steps. With the jump, each one costs one line of arithmetic: pick a data point, a step t and a noise draw, and compute directly, with no need to run the t steps before it.
The maths
With ,
that is, with one draw . Why: suppose it holds at step . Then , and the two noise terms are independent with mean 0, so their sum is Gaussian with variance on each axis. At the last step, 0.0045, so keeps 0.067 of .
-
The network's job: guess the noise
The network is shown a noisy point and its step t, and asked: what noise was added to get here?
On the left, one data point (the square) and its noisy version at step t (the cyan dot). On the right, the noise that was really drawn, (dashed), and the network's guess, (solid); the dotted line between their tips is the error. Until the network is trained in the next step, its guesses come from random starting weights. Come back after training to see the trained network's guess.
The network is small. It takes the point's x and y, together with wiggly versions of x and y (sines and cosines of them), which help it draw fine detail, and the step t in the same way. Two hidden layers of 64 ReLU units (from hidden layers) lead to two outputs, the guess of the noise on each axis: 5,698 weights and biases in all.
Its loss is the squared error between the true and the guessed noise, averaged over a batch: the mean squared error of measuring error, with the squared errors on the two axes added for each row before the average is taken. It cannot reach 0. The network sees only the noisy point and its step t, and many pairs of data point and noise could have led to that same noisy point. The best it can do is a weighted average of their noises, where the pairs more likely to have made that point count for more.
The maths
The network , with weights , is trained to make
small, with , where the expectation is over a data point , a step t drawn evenly from 1 to 50, and . In training, the expectation is replaced by the mean over a batch of 128 such rows. The guess that makes this smallest is the average of given and t, which is not the drawn itself, so the loss stays above 0.
The inputs are 21 numbers: x, y, and for (14 in all), then and for (7). The count: .
-
Train it
Press Train. The curve is the loss on a fixed set of 512 rows, checked every 100 steps.
Play steps through the lesson, and Train trains the network. Each training step draws a fresh batch of 128 rows (a data point, a step t and a noise draw each) and takes one step of Adam (from optimisers) with η = 0.003. The run is 6,000 steps: about half a minute on a desktop computer, and a minute or two on a phone.
The dashed line is the loss of always guessing 0, which on these rows is 1.95, close to 2, the average of for two standard Gaussian numbers. The starting network does worse than that: 2.70. After 100 steps the loss is 0.82, and by step 6,000 it has crept down to 0.73, where it levels off, well above 0, for the reason in the previous step.
Reset goes back to the same starting weights, and the next run is the same as the last. Real systems use the same idea at far larger scale.
The maths
The fixed 512 rows are drawn once, like a training batch, and never used for training, so the curve is not jumpy the way single batches are. The network starts with random weights: each weight a normal draw with mean 0 and standard deviation , where is the number of inputs to its layer, and every bias 0.
-
Generate: from noise to a spiral
Press Generate. 400 points of pure noise walk backwards from t = 50 to t = 0, and faint trails show the way they came.
Generation starts where the noise recipe ends: 400 draws from N(0, I), which is very nearly what any data look like at t = 50. Each step back asks the network for its noise guess, takes out a share of it, undoes the shrink, and adds a little fresh noise (except on the last step). Fifty such steps later the points lie along the two arms. Generate uses the network as it is now, so train it in the previous step first. Each press starts from new noise.
Two numbers judge the new points, as in generative models: how far each new point is from its nearest data point, and how far each data point is from its nearest new point. Call the first precision and the second coverage; smaller is closer for both. Both depend on how many points there are, so every set compared here has 400 points, measured against the same 400 data points.
After the full 6,000-step run, for the first press, the starting noise scores 0.22 and 0.11: many noise points fall between the arms, though some lie near every part of the spiral. The mixture of five Gaussians from generative models scores 0.15 and 0.09. The diffusion samples score 0.08 and 0.06, closer both ways; in each of the first 20 presses they are closer than both the mixture and the noise, both ways. Fresh points from the spiral's own rule score 0.05 and 0.06. The samples are a little farther from the data than fresh points are, so some land just off the arms; and they are not copies of the data, which would score 0.
The maths
Start from and, for ,
with for and for . This is Algorithm 2 of Ho, Jain & Abbeel (2020). here is the standard deviation of the fresh noise added at each step, not the sigmoid. The two numbers, for samples and data , are
with throughout.
-
What real systems add
Diffusion models that make pictures use the same three steps: add noise, learn to guess it, remove it step by step.
Compared with this page, they use far bigger networks, with far more weights, and images instead of points: a 64 × 64 colour image is a list of 64 × 64 × 3 = 12,288 numbers, so it is one point in 12,288 dimensions. Many also take a text description as an extra input to the network, so that the noise it removes steers the image towards what the text describes.
The recipe on this page is the one in Ho, Jain & Abbeel (2020). Later systems change details, such as what the network is trained to predict, or adding the noise to a compressed version of the image rather than to its pixels, but they keep the three steps: a way of adding noise, a network trained to undo it, and new data that start as pure noise.
Quick check
Lesson complete
- A fixed recipe adds a little Gaussian noise many times, until the data are nearly pure noise; one formula jumps straight to any step.
- A network is trained to guess the added noise from the noisy point and its step, with the squared error as the loss.
- New data start as pure noise, and each step back takes out part of the guessed noise. Here the 400 new points were closer to the 400 data points than the mixture of five Gaussians, both ways.
Common misconceptions
- "Diffusion copies the training points." The samples sat 0.08 from their nearest data point on average; copies would sit at 0.
- "The network removes all the noise at once." It is used 50 times; each step takes out part of the guessed noise and, except the last, adds a little fresh noise.
- "The noise is learned." The forward noise is a fixed recipe. Only the noise-guessing network is trained.
That is the end of Level 5. Level 6 is coming next.
Back to the path →Further reading
- Prince, Understanding Deep Learning (2023), Chapter 18, "Diffusion models".
- Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics", arXiv 1503.03585 (2015), which trains a model to reverse a process that slowly turns data into noise.
- Ho, Jain & Abbeel, "Denoising Diffusion Probabilistic Models", arXiv 2006.11239 (2020), whose Algorithm 2 is the step back used here, and whose training loss compares the true and the guessed noise.
About the data. All the data is made in your browser by the mulberry32 random number generator, with Gaussian draws by the Box–Muller method. The spiral, as in generative models: 400 points from seed 305, 200 on each of two arms; along an arm the radius grows from 0.3 towards 3 over one turn, the second arm starts half a turn round from the first, and Gaussian noise with standard deviation 0.1 is added to each coordinate; each axis is then standardised to mean 0 and standard deviation 1 (dividing by n). The fresh spiral is made the same way from seed 307. The noise schedule is βt from 0.001 to 0.2 in equal steps over T = 50, a short schedule chosen for this page. The slider's noise is one pair of normal draws per point, in order, from seed 300. The network's starting weights use seed 311. Training uses seed 312: each row draws a data point (evenly, with replacement), a step t evenly from 1 to 50, and two normal draws for the noise; Adam with η = 0.003 and Adam's own settings β₁ = 0.9, β₂ = 0.999 (not the noise schedule) and a small constant of 10⁻⁸, batches of 128, 6,000 steps. The fixed 512 rows for the loss curve are drawn the same way from seed 313 (the same seed as press 1 of Generate; the two draw different things). Press n of Generate starts from seed 313 + n − 1: first 400 pairs of normal draws, then a pair per point at every step but the last. The mixture is the K = 5 fit of generative models: EM from seed 304, stopped when the mean log-likelihood rises by less than 10⁻⁶ or after 500 iterations; for press n, its 400 points use seed 310 + n − 1.
If this helped, you can .