Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Backpropagation

How did the network know which way to nudge each weight? Follow every number through a tiny network, forward and then backward.

About 12 minutes · Some arithmetic on the stage; the calculus is in The maths

Overview

In the previous lesson a network adjusted its weights, a small step at a time, to lower its loss. Each step needs to know, for every weight, which way to move and how strongly the loss responds: the gradient, exactly as in gradient descent. Backpropagation is how those gradients are worked out.

This lesson does it on a network small enough to show every number: two inputs, two hidden units and one output, with one example. The weights are chosen by hand, one decimal place each, so the arithmetic is easy to follow. Values flow forward to give a prediction and a loss; then errors flow backward to give every weight's gradient in one sweep.

The example and the weights are made up. See "About the data" at the end.

You'll learn

  • how the forward pass turns an input into a prediction and a loss
  • how the chain rule, applied from the output backwards, gives every weight's gradient in one sweep
  • how to check a gradient by nudging a weight, and how a training step uses it
  1. Forward pass

    Values fill the diagram from left to right. Each unit adds up its inputs times their weights, adds its bias, and passes the total through the sigmoid.

    The example is x = (1.0, −0.5). Each link carries its weight: solid cyan for positive, dashed violet for negative, thicker for larger. The biases are in the read-out.

    Hidden unit h₁ works out z = 0.5 × 1.0 + (−0.4) × (−0.5) + 0.1 = 0.800, then its output a = σ(z) = 0.690. Hidden unit h₂ gets z = −0.900 and a = 0.289.

    The output unit does the same with the two hidden outputs: z = 0.510, and p^\hat{p} = σ(z) = 0.625 is the network's probability that the example is class 1. Numbers are shown to 3 significant figures; the sums use the full values.

    The maths

    With layers numbered as on the notation page, a(0)=x\mathbf{a}^{(0)} = \mathbf{x} and

    z(1)=W(1)x+b(1),a(1)=σ(z(1)),z(2)=W(2)a(1)+b(2),p^=σ(z(2))\mathbf{z}^{(1)} = \mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}, \quad \mathbf{a}^{(1)} = \sigma\big(\mathbf{z}^{(1)}\big), \quad z^{(2)} = \mathbf{W}^{(2)}\mathbf{a}^{(1)} + b^{(2)}, \quad \hat{p} = \sigma\big(z^{(2)}\big)

    where σ(z)=1/(1+e−z)\sigma(z) = 1/(1 + e^{-z}) acts on each entry. σ here is the sigmoid, not a standard deviation. The hand-chosen weights are

    W(1)=(0.5−0.4−0.30.8),b(1)=(0.1−0.2),W(2)=(0.7−0.6),b(2)=0.2\mathbf{W}^{(1)} = \begin{pmatrix} 0.5 & -0.4 \\ -0.3 & 0.8 \end{pmatrix}, \quad \mathbf{b}^{(1)} = \begin{pmatrix} 0.1 \\ -0.2 \end{pmatrix}, \quad \mathbf{W}^{(2)} = \begin{pmatrix} 0.7 & -0.6 \end{pmatrix}, \quad b^{(2)} = 0.2

    Row jj of W(1)\mathbf{W}^{(1)} holds the weights into hidden unit jj.

  2. The loss

    The label is y = 1, so the loss is small when p^\hat{p} is close to 1 and grows as it falls away.

    The loss ℒ is binary cross-entropy, the log-loss of logistic regression, here for one example. With y = 1 it is ℒ = −ln p^\hat{p} = −ln 0.625 = 0.470.

    If p^\hat{p} were 1 the loss would be 0. Training wants this number lower, so the question for every weight is: which way should it move, and how strongly does the loss respond?

    The maths
    L=−[ yln⁡p^+(1−y)ln⁡(1−p^)]\mathcal{L} = -\big[\, y \ln \hat{p} + (1 - y)\ln(1 - \hat{p}) \big]

    The page computes it from the output's logit z(2)z^{(2)} as ln⁡ ⁣(1+ez(2))−y z(2)\ln\!\big(1 + e^{z^{(2)}}\big) - y\,z^{(2)}, the same quantity written so it stays accurate when p^\hat{p} is very close to 0 or 1. Over a batch of examples, ℒ is the mean of the per-example losses.

  3. Output error

    Start at the end. How much does the loss change when the output's z changes? For a sigmoid output with this loss the answer is simply p^−y\hat{p} - y.

    That number is the output's error, δ⁽²⁾ = p^−y\hat{p} - y = 0.625 − 1 = −0.375, drawn in violet beside the output.

    Its sign says which way z should move: negative, so raising z would lower the loss. Its size says how strongly the loss responds to a small change in z. The derivation is a few lines, in The maths.

    The maths

    By definition δ(2)=∂L/∂z(2)\delta^{(2)} = \partial\mathcal{L}/\partial z^{(2)}. By the chain rule, through p^=σ(z(2))\hat{p} = \sigma(z^{(2)}):

    ∂L∂z(2)=∂L∂p^⋅dp^dz(2)=p^−yp^(1−p^)⋅p^(1−p^)=p^−y\frac{\partial\mathcal{L}}{\partial z^{(2)}} = \frac{\partial\mathcal{L}}{\partial \hat{p}}\cdot\frac{d\hat{p}}{dz^{(2)}} = \frac{\hat{p} - y}{\hat{p}(1 - \hat{p})}\cdot \hat{p}(1 - \hat{p}) = \hat{p} - y

    using ∂L/∂p^=−y/p^+(1−y)/(1−p^)\partial\mathcal{L}/\partial\hat{p} = -y/\hat{p} + (1 - y)/(1 - \hat{p}) and the sigmoid's slope σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big). The two factors cancel, which is why this pairing of sigmoid and log-loss is so common. The same p^−y\hat{p} - y appears in the gradient of logistic regression's log-loss.

  4. Backwards through a layer

    Each hidden unit's error is the output error sent back along its weight, scaled by the sigmoid's slope at that unit. Each weight's gradient is then the δ at its output end times the a at its input end.

    Hidden unit h₁ feeds the output through the weight 0.7, so its error is δ = 0.7 × (−0.375) × σ′(0.800) = −0.0562. For h₂ it is δ = 0.0463. The errors fill from right to left, the opposite way to the forward pass.

    Choose a weight: point at or tab to a link, or use the weight menu. Its gradient ∂ℒ/∂w appears on the link and the read-out shows the product. For the weight from h₁ to the output it is −0.375 × 0.690 = −0.259; for the weight from x₁ to h₁ it is −0.0562 × 1.00 = −0.0562.

    None of the six gradients is 0 here, so every weight has a direction to move. Each agrees with a careful numerical estimate to better than one part in a million; the next step shows the idea behind that check.

    The maths

    A weight ww into a unit only affects the loss through that unit's zz, so by the chain rule

    ∂L∂w=∂L∂z⋅∂z∂w=δ⋅ain\frac{\partial\mathcal{L}}{\partial w} = \frac{\partial\mathcal{L}}{\partial z}\cdot\frac{\partial z}{\partial w} = \delta \cdot a_{\text{in}}

    because zz is a sum of weights times inputs, and ∂z/∂w\partial z/\partial w is the input aina_{\text{in}} on that link. A bias's gradient is the unit's δ alone. The hidden errors come from the one after them:

    δ(1)=(W(2)Tδ(2))⊙σ′(z(1))\boldsymbol{\delta}^{(1)} = \big(\mathbf{W}^{(2)\mathsf{T}}\delta^{(2)}\big) \odot \sigma'\big(\mathbf{z}^{(1)}\big)

    where ⊙ multiplies entry by entry. In a deeper network the same rule repeats, δ(ℓ)=(W(ℓ+1)Tδ(ℓ+1))⊙g′(z(ℓ))\boldsymbol{\delta}^{(\ell)} = \big(\mathbf{W}^{(\ell+1)\mathsf{T}}\boldsymbol{\delta}^{(\ell+1)}\big) \odot g'\big(\mathbf{z}^{(\ell)}\big), layer by layer from the output back to the first layer. Each δ is reused by every weight into its unit, which is what makes one backward sweep enough.

  5. Check it

    A gradient is a prediction: nudge the weight a little and the loss should change by about gradient × nudge. Try it.

    Choose a weight, then press +0.01 or −0.01. The diagram recomputes every value with the nudged weight. The read-out shows, for both nudges, the real change in the loss, from running the forward pass again, beside the gradient × 0.01.

    For the weight from h₁ to the output, +0.01 changes the loss by −0.00258, against a prediction of −0.00259. For all six weights and both nudges the two agree to within 2%. They are not exactly equal because the loss curves; the gap shrinks as the nudge shrinks.

    Nudging each weight in turn is another way to find gradients. Backpropagation gets all of them from one sweep instead; The maths compares the cost.

    The maths

    The nudge test is a finite difference:

    ∂L∂w≈L(w+h)−L(w)h\frac{\partial\mathcal{L}}{\partial w} \approx \frac{\mathcal{L}(w + h) - \mathcal{L}(w)}{h}

    with h=0.01h = 0.01 here. Its error shrinks in proportion to hh; the central version [L(w+h)−L(w−h)]/(2h)\big[\mathcal{L}(w + h) - \mathcal{L}(w - h)\big]/(2h) is more accurate and is the one used to check the gradients on this page.

    Cost: finding every gradient by nudging needs at least one extra forward pass per weight, so a network with a million weights would need a million forward passes for one gradient. Backpropagation needs one forward pass and one backward pass, and as a rough rule the backward pass costs a small multiple of a forward pass, however many weights there are. Real systems compute these gradients automatically, applying the same chain rule to each operation in turn.

  6. One training step, then many

    Move every weight against its gradient by η times the gradient, and the loss drops. Repeat on a set of points and the network learns them.

    With learning rate η = 3, one step on the example moves all nine numbers at once, the six weights and three biases, and the loss falls from 0.470 to 0.0875. The read-out shows it.

    Now press Train: 200 steps of plain gradient descent on 20 XOR points, the same kind of four-clump data as in the artificial neuron lesson, starting from the same hand-chosen weights. Each step runs the forward and backward pass for every point, averages the gradients and moves every weight. Play steps through the lesson, and Train trains the network. Moving to another step pauses training.

    On this seeded run (the end-of-run numbers here are for the full 200-step run; the read-out says finished once the run is done) the mean loss falls from 0.766 to 0.125, lower at every one of the 200 steps, and the training accuracy rises from 20.0% to 100%, first reaching 100% at step 174. That is on the 20 training points only; it says nothing yet about new points.

    "Learns" here means exactly that: adjusts its weights to lower the loss on the training data. This run uses plain gradient descent rather than the previous lesson's Adam, so each step is exactly η times the gradient; the curve on the left is the mean loss at the start and after each step.

    The maths

    Every weight and bias takes the same kind of step:

    w←w−η ∂L∂ww \leftarrow w - \eta\,\frac{\partial\mathcal{L}}{\partial w}

    All the gradients are found first, from the same weights, and then every weight moves together. On the XOR points ℒ is the mean loss over the 20 points, and its gradient is the mean of the per-example gradients, each found by one forward and one backward pass. A learning rate of 3 is large; on this run it works, since the loss falls at every step, but a different network or data could need a smaller one.

Quick check

Why go backwards rather than forwards?

Right. The loss is at the end, and each δ needs the δ after it. Going backwards, every δ is computed once and reused by all the weights into its unit, so, as a rough rule, one sweep costs a small multiple of a forward pass.

The gradient is the same number however it is worked out; the nudge check in step 5 finds nearly the same values without going backwards at all. Going backwards is about cost: one sweep gives all of them.

The order is not about importance. The output's weights come first only because the hidden errors need δ⁽²⁾ before they can be worked out.

On this page every gradient is found first, from the same weights, and then all the weights move together in one step, as in step 6. The backward order comes from each layer's δ needing the δ of the layer after it, not from updating as the error arrives.

If δ at a node is 0, what happens to the gradients of the weights coming into it?

Right. Each incoming weight's gradient is δ × a, so with δ = 0 every one is 0, and so is the bias's. For this example those weights would not move; over a batch, other examples can still move them.

A gradient is δ × a for that link, not the weight. With δ = 0 the product is 0 whatever the weight is.

The opposite: δ × a with δ = 0 is 0, so those weights get no push at all from this example.

The bias's gradient is δ, which is 0, but each weight's gradient is δ × a, which is 0 too.

Lesson complete

  • The forward pass turns the input into p^\hat{p} and a loss; the output's error is p^−y\hat{p} - y.
  • The chain rule sends the error backwards one layer at a time, and each weight's gradient is the δ at its output end times the a at its input end.
  • A gradient predicts how the loss responds to a small nudge, and a training step moves every weight by η times its gradient.

Common misconceptions

  • "Backpropagation is the learning." It only finds the gradients. The step that changes the weights is gradient descent (or another optimiser): on this page, step 4 finds the gradients and step 6 uses them.
  • "The gradient says how far to move each weight." It says which way to move and how strongly the loss responds near the current weights. The nudges of 0.01 in step 5 matched it to within 2%; how far each step goes is set by η.
  • "Backpropagation gives approximate gradients, like nudging." It gives the exact gradient, up to computer rounding. Here it agrees with a careful central-difference estimate to better than one part in a million; it is the nudging that is approximate.
Next lesson: Training a network →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 6, Section 6.5, "Back-Propagation and Other Differentiation Algorithms", which builds backpropagation from computational graphs and the chain rule.
  2. Bishop, Pattern Recognition and Machine Learning, its chapter on neural networks, which derives error backpropagation.

About the data. The one example, x = (1.0, −0.5) with label y = 1, and the weights in step 1's maths are chosen by hand, one decimal place each, so that no gradient is 0. The 20 XOR points in step 6 are made up in your browser by the mulberry32 random number generator with seed 161: each point picks one of the corners (±1, ±1) at random, each equally likely, and adds 0.35 z to each coordinate, where z is a draw from a normal distribution with mean 0 and SD 1. A point is class 1 when its corner's two signs differ. The run uses plain gradient descent with η = 3, every step on all 20 points, for 200 steps, from the same hand-chosen weights. The trainer's shuffle uses seed 162, but every step uses all 20 points, so the shuffle only changes the order of a sum and the results differ at most by rounding.

If this helped, you can .