ML Explorer · Level 4 · Deep learning
Backpropagation
How did the network know which way to nudge each weight? Follow every number through a tiny network, forward and then backward.
Overview
In the previous lesson a network adjusted its weights, a small step at a time, to lower its loss. Each step needs to know, for every weight, which way to move and how strongly the loss responds: the gradient, exactly as in gradient descent. Backpropagation is how those gradients are worked out.
This lesson does it on a network small enough to show every number: two inputs, two hidden units and one output, with one example. The weights are chosen by hand, one decimal place each, so the arithmetic is easy to follow. Values flow forward to give a prediction and a loss; then errors flow backward to give every weight's gradient in one sweep.
The example and the weights are made up. See "About the data" at the end.
You'll learn
- how the forward pass turns an input into a prediction and a loss
- how the chain rule, applied from the output backwards, gives every weight's gradient in one sweep
- how to check a gradient by nudging a weight, and how a training step uses it
-
Forward pass
Values fill the diagram from left to right. Each unit adds up its inputs times their weights, adds its bias, and passes the total through the sigmoid.
The example is x = (1.0, −0.5). Each link carries its weight: solid cyan for positive, dashed violet for negative, thicker for larger. The biases are in the read-out.
Hidden unit h₁ works out z = 0.5 × 1.0 + (−0.4) × (−0.5) + 0.1 = 0.800, then its output a = σ(z) = 0.690. Hidden unit h₂ gets z = −0.900 and a = 0.289.
The output unit does the same with the two hidden outputs: z = 0.510, and = σ(z) = 0.625 is the network's probability that the example is class 1. Numbers are shown to 3 significant figures; the sums use the full values.
The maths
With layers numbered as on the notation page, and
where acts on each entry. σ here is the sigmoid, not a standard deviation. The hand-chosen weights are
Row of holds the weights into hidden unit .
-
The loss
The label is y = 1, so the loss is small when is close to 1 and grows as it falls away.
The loss ℒ is binary cross-entropy, the log-loss of logistic regression, here for one example. With y = 1 it is ℒ = −ln = −ln 0.625 = 0.470.
If were 1 the loss would be 0. Training wants this number lower, so the question for every weight is: which way should it move, and how strongly does the loss respond?
The maths
The page computes it from the output's logit as , the same quantity written so it stays accurate when is very close to 0 or 1. Over a batch of examples, ℒ is the mean of the per-example losses.
-
Output error
Start at the end. How much does the loss change when the output's z changes? For a sigmoid output with this loss the answer is simply .
That number is the output's error, δ⁽²⁾ = = 0.625 − 1 = −0.375, drawn in violet beside the output.
Its sign says which way z should move: negative, so raising z would lower the loss. Its size says how strongly the loss responds to a small change in z. The derivation is a few lines, in The maths.
The maths
By definition . By the chain rule, through :
using and the sigmoid's slope . The two factors cancel, which is why this pairing of sigmoid and log-loss is so common. The same appears in the gradient of logistic regression's log-loss.
-
Backwards through a layer
Each hidden unit's error is the output error sent back along its weight, scaled by the sigmoid's slope at that unit. Each weight's gradient is then the δ at its output end times the a at its input end.
Hidden unit h₁ feeds the output through the weight 0.7, so its error is δ = 0.7 × (−0.375) × σ′(0.800) = −0.0562. For h₂ it is δ = 0.0463. The errors fill from right to left, the opposite way to the forward pass.
Choose a weight: point at or tab to a link, or use the weight menu. Its gradient ∂ℒ/∂w appears on the link and the read-out shows the product. For the weight from h₁ to the output it is −0.375 × 0.690 = −0.259; for the weight from x₁ to h₁ it is −0.0562 × 1.00 = −0.0562.
None of the six gradients is 0 here, so every weight has a direction to move. Each agrees with a careful numerical estimate to better than one part in a million; the next step shows the idea behind that check.
The maths
A weight into a unit only affects the loss through that unit's , so by the chain rule
because is a sum of weights times inputs, and is the input on that link. A bias's gradient is the unit's δ alone. The hidden errors come from the one after them:
where ⊙ multiplies entry by entry. In a deeper network the same rule repeats, , layer by layer from the output back to the first layer. Each δ is reused by every weight into its unit, which is what makes one backward sweep enough.
-
Check it
A gradient is a prediction: nudge the weight a little and the loss should change by about gradient × nudge. Try it.
Choose a weight, then press +0.01 or −0.01. The diagram recomputes every value with the nudged weight. The read-out shows, for both nudges, the real change in the loss, from running the forward pass again, beside the gradient × 0.01.
For the weight from h₁ to the output, +0.01 changes the loss by −0.00258, against a prediction of −0.00259. For all six weights and both nudges the two agree to within 2%. They are not exactly equal because the loss curves; the gap shrinks as the nudge shrinks.
Nudging each weight in turn is another way to find gradients. Backpropagation gets all of them from one sweep instead; The maths compares the cost.
The maths
The nudge test is a finite difference:
with here. Its error shrinks in proportion to ; the central version is more accurate and is the one used to check the gradients on this page.
Cost: finding every gradient by nudging needs at least one extra forward pass per weight, so a network with a million weights would need a million forward passes for one gradient. Backpropagation needs one forward pass and one backward pass, and as a rough rule the backward pass costs a small multiple of a forward pass, however many weights there are. Real systems compute these gradients automatically, applying the same chain rule to each operation in turn.
-
One training step, then many
Move every weight against its gradient by η times the gradient, and the loss drops. Repeat on a set of points and the network learns them.
With learning rate η = 3, one step on the example moves all nine numbers at once, the six weights and three biases, and the loss falls from 0.470 to 0.0875. The read-out shows it.
Now press Train: 200 steps of plain gradient descent on 20 XOR points, the same kind of four-clump data as in the artificial neuron lesson, starting from the same hand-chosen weights. Each step runs the forward and backward pass for every point, averages the gradients and moves every weight. Play steps through the lesson, and Train trains the network. Moving to another step pauses training.
On this seeded run (the end-of-run numbers here are for the full 200-step run; the read-out says finished once the run is done) the mean loss falls from 0.766 to 0.125, lower at every one of the 200 steps, and the training accuracy rises from 20.0% to 100%, first reaching 100% at step 174. That is on the 20 training points only; it says nothing yet about new points.
"Learns" here means exactly that: adjusts its weights to lower the loss on the training data. This run uses plain gradient descent rather than the previous lesson's Adam, so each step is exactly η times the gradient; the curve on the left is the mean loss at the start and after each step.
The maths
Every weight and bias takes the same kind of step:
All the gradients are found first, from the same weights, and then every weight moves together. On the XOR points ℒ is the mean loss over the 20 points, and its gradient is the mean of the per-example gradients, each found by one forward and one backward pass. A learning rate of 3 is large; on this run it works, since the loss falls at every step, but a different network or data could need a smaller one.
Quick check
Lesson complete
- The forward pass turns the input into and a loss; the output's error is .
- The chain rule sends the error backwards one layer at a time, and each weight's gradient is the δ at its output end times the a at its input end.
- A gradient predicts how the loss responds to a small nudge, and a training step moves every weight by η times its gradient.
Common misconceptions
- "Backpropagation is the learning." It only finds the gradients. The step that changes the weights is gradient descent (or another optimiser): on this page, step 4 finds the gradients and step 6 uses them.
- "The gradient says how far to move each weight." It says which way to move and how strongly the loss responds near the current weights. The nudges of 0.01 in step 5 matched it to within 2%; how far each step goes is set by η.
- "Backpropagation gives approximate gradients, like nudging." It gives the exact gradient, up to computer rounding. Here it agrees with a careful central-difference estimate to better than one part in a million; it is the nudging that is approximate.
Further reading
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 6, Section 6.5, "Back-Propagation and Other Differentiation Algorithms", which builds backpropagation from computational graphs and the chain rule.
- Bishop, Pattern Recognition and Machine Learning, its chapter on neural networks, which derives error backpropagation.
About the data. The one example, x = (1.0, −0.5) with label y = 1, and the weights in step 1's maths are chosen by hand, one decimal place each, so that no gradient is 0. The 20 XOR points in step 6 are made up in your browser by the mulberry32 random number generator with seed 161: each point picks one of the corners (±1, ±1) at random, each equally likely, and adds 0.35 z to each coordinate, where z is a draw from a normal distribution with mean 0 and SD 1. A point is class 1 when its corner's two signs differ. The run uses plain gradient descent with η = 3, every step on all 20 points, for 200 steps, from the same hand-chosen weights. The trainer's shuffle uses seed 162, but every step uses all 20 points, so the shuffle only changes the order of a sum and the results differ at most by rounding.
If this helped, you can .