Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Going deeper: vanishing gradients

Stack ten hidden layers of sigmoid units and the gradient that reaches the first layer can be millions of times smaller than at the last. Here you measure it layer by layer, see why, and watch a ReLU network with better starting weights train where the sigmoid one stalls.

About 10 minutes · Every bar and curve is computed in your browser; the formulas are in The maths

Overview

The networks so far had one hidden layer. Adding more lets a network build features out of features, but it makes training harder. Backpropagation passes the error signal backwards one layer at a time, multiplying it by something at every layer. Over many layers those factors pile up.

This lesson takes a network with ten hidden layers, measures the size of the gradient for each layer's weights at the start of training, and shows how the choice of activation and of starting weights changes it. Then it trains two deep networks side by side.

The data are two rings like those in hidden layers, made up in your browser; see "About the data" at the end.

You'll learn

  • why, in a deep sigmoid network, the gradient for the first layers can be far smaller than for the last
  • how ReLU with He starting weights keeps the gradient about the same size from layer to layer here
  • what Xavier and He initialisation aim for, and what they assume
  1. Stack more layers

    Two inputs, ten hidden layers of 8 units, one output: 681 weights and biases.

    Each hidden layer takes the 8 values of the layer before, forms 8 weighted sums plus biases, and passes each through the same activation g. There are 11 layers of weights, numbered ℓ = 1 (out of the inputs) to ℓ = 11 (into the output). In steps 2 and 3 every hidden unit is a sigmoid. σ here is the sigmoid, not a standard deviation.

    One hidden layer would be plenty for these rings. The extra depth is here to show what depth does to training, not to win accuracy.

    The maths

    For layers ℓ=1,…,L\ell = 1, \ldots, L with L=11L = 11 and a(0)=x\mathbf{a}^{(0)} = \mathbf{x}:

    z(ℓ)=W(ℓ)a(ℓ−1)+b(ℓ),a(ℓ)=g(z(ℓ))\mathbf{z}^{(\ell)} = \mathbf{W}^{(\ell)}\mathbf{a}^{(\ell-1)} + \mathbf{b}^{(\ell)}, \qquad \mathbf{a}^{(\ell)} = g\big(\mathbf{z}^{(\ell)}\big)

    for the hidden layers ℓ=1,…,10\ell = 1, \ldots, 10. The output z(11)z^{(11)} is a single number with no activation; its sigmoid is the predicted probability of class 1. The count is 8(2+1)+9×8(8+1)+1(8+1)=6818(2+1) + 9 \times 8(8+1) + 1(8+1) = 681.

  2. Gradient size per layer, at the start

    One bar per layer of weights: how big the gradient for that layer is before any training. The chart uses a log scale: each grid line is 100 times the one below.

    With sigmoid units and Xavier starting weights, the gradient for layer 11 has size 0.14. Each layer back it is smaller, and at layer 1 it is 1.2×10⁻⁸: 8.7×10⁻⁸ times layer 11's, or about 11 million times smaller.

    Plain gradient descent moves each weight by η times its gradient, so with one η for the whole network the first layers barely move while the last ones do. On the log scale the bars climb like a staircase from left to right.

    The maths

    ℒ is the mean loss over all 200 training points, each point's loss being binary cross-entropy, the log-loss of logistic regression. Each bar is the Frobenius norm of the gradient for one weight matrix, the square root of the sum of its squared entries:

    ∥∂L∂W(ℓ)∥=∑j∑i(∂L∂Wji(ℓ))2\left\lVert \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(\ell)}} \right\rVert = \sqrt{\sum_{j}\sum_{i} \left(\frac{\partial \mathcal{L}}{\partial W^{(\ell)}_{ji}}\right)^{2}}

    computed by backpropagation at the starting weights. The biases are left out of the bars.

  3. Why it shrinks

    On the way back, every sigmoid layer multiplies the error signal by its slope σ′, and that slope is never more than 1/4.

    Backpropagation gets layer ℓ's error signal from layer ℓ + 1's: multiply by layer ℓ + 1's weights, transposed, then by the slope of the activation at each unit of layer ℓ. For the sigmoid that slope is at most 1/4, reached only at z = 0 (the inset).

    Xavier starting weights are scaled so that, on average, multiplying by the weights keeps the signal about the same size. So the slopes decide. At the start the sigmoid units sit near their steepest: averaged over the training points, the slope in each hidden layer is between 0.22 and 0.24. Even so, each factor is under a quarter, and on average the gradient size falls by a factor of 5.1 per layer from layer 11 back to layer 1.

    That is this network with these starting weights. Whether the gradient shrinks, holds or grows depends on the activation and on the size of the weights, which is what the next step lets you change.

    The maths

    Here, for one training point, ℒ is that point's loss (the bars average the resulting gradients over the 200 points). With δ(ℓ)=∂L/∂z(ℓ)\boldsymbol{\delta}^{(\ell)} = \partial\mathcal{L}/\partial\mathbf{z}^{(\ell)}, backpropagation gives

    δ(ℓ)=(W(ℓ+1)⊤δ(ℓ+1))⊙g′(z(ℓ)),∂L∂W(ℓ)=δ(ℓ) a(ℓ−1)⊤\boldsymbol{\delta}^{(\ell)} = \Big(\mathbf{W}^{(\ell+1)\top}\boldsymbol{\delta}^{(\ell+1)}\Big) \odot g'\big(\mathbf{z}^{(\ell)}\big), \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(\ell)}} = \boldsymbol{\delta}^{(\ell)}\,\mathbf{a}^{(\ell-1)\top}

    where ⊙ multiplies entry by entry, starting from δ(11)=σ(z(11))−y\delta^{(11)} = \sigma(z^{(11)}) - y. Unrolled, the error signal at layer ℓ is a product of one weight matrix and one set of slopes for every layer above it:

    δ(ℓ)=D(ℓ)W(ℓ+1)⊤D(ℓ+1)W(ℓ+2)⊤⋯D(10)W(11)⊤δ(11),D(k)=diag⁡(g′(z(k)))\boldsymbol{\delta}^{(\ell)} = \mathbf{D}^{(\ell)}\mathbf{W}^{(\ell+1)\top}\mathbf{D}^{(\ell+1)}\mathbf{W}^{(\ell+2)\top}\cdots\mathbf{D}^{(10)}\mathbf{W}^{(11)\top}\delta^{(11)}, \qquad \mathbf{D}^{(k)} = \operatorname{diag}\big(g'(\mathbf{z}^{(k)})\big)

    For the sigmoid, σ′(z)=σ(z)(1−σ(z))=14−(σ(z)−12)2≤14\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big) = \tfrac14 - \big(\sigma(z) - \tfrac12\big)^2 \le \tfrac14, with equality exactly when σ(z)=12\sigma(z) = \tfrac12, that is z=0z = 0. So each step back obeys

    ∥δ(ℓ)∥≤14 ∥W(ℓ+1)∥2 ∥δ(ℓ+1)∥\big\lVert\boldsymbol{\delta}^{(\ell)}\big\rVert \le \tfrac14\,\big\lVert\mathbf{W}^{(\ell+1)}\big\rVert_2\,\big\lVert\boldsymbol{\delta}^{(\ell+1)}\big\rVert

    where ∥W∥2\lVert\mathbf{W}\rVert_2 is the most that W⊤\mathbf{W}^\top can stretch a vector (the same for W\mathbf{W}). Across ten sigmoid layers the slopes alone contribute at most (1/4)10≈9.5×10−7(1/4)^{10} \approx 9.5 \times 10^{-7}; unless the weights' stretches multiply to more than 4104^{10}, the bound shrinks. It is a bound, not the actual size, which is why the page measures the bars instead. The step's factor of 5.1 per layer is the tenth root of the measured ratio, (∥∂L/∂W(11)∥/∥∂L/∂W(1)∥)1/10(\lVert\partial\mathcal{L}/\partial\mathbf{W}^{(11)}\rVert / \lVert\partial\mathcal{L}/\partial\mathbf{W}^{(1)}\rVert)^{1/10}, an average over ten steps back; the steps differ, and the largest is the last one, from layer 2 to layer 1, a factor of about 22. For tanh the slope 1−tanh⁡2z1 - \tanh^2 z is at most 1, and for ReLU it is 0 or 1.

  4. Activation and starting weights

    Pick the activation and the size of the starting weights; the bars are recomputed for the same network and data.

    "Xavier / He" means Xavier for sigmoid and tanh and He for ReLU. "sd 0.1" and "sd 2" draw every weight with that standard deviation, whatever the layer. All settings use the same random draws, only scaled, and every bias starts at 0.

    ReLU with He: every bar is between 0.1 and 10, and layer 1's is 0.47 times layer 11's. Tanh with Xavier holds steady too, with a ratio of 0.42; tanh's slope near 0 is 1, not 1/4.

    Too small, sd 0.1: with ReLU or tanh every bar is below 10⁻⁶. Small weights shrink the signal on the way forward and the error signal on the way back, so every layer gets almost nothing. With the sigmoid, layer 1's bar drops below 10⁻¹².

    Too large, sd 2: with ReLU every bar shoots past the top of the chart; the largest is 2.0×10⁶, against 3.3 with He. Bars that pass 10² are cut off and marked with a triangle. With tanh, layer 1's gradient is 320 times layer 11's. With the sigmoid, sd 2 helps: the ratio is 5.0×10⁻³, still under a hundredth.

    The maths

    Take one unit, zj=∑i=1ninWji aiz_j = \sum_{i=1}^{n_{\text{in}}} W_{ji}\,a_i, and assume the weights are independent, with mean 0 and variance vv, and independent of the inputs aia_i, which are themselves independent of each other with the same distribution. Then

    Var⁡(zj)=nin v E[ai2]\operatorname{Var}(z_j) = n_{\text{in}}\, v\, \mathbb{E}[a_i^2]

    Xavier initialisation, in the version used here (also called LeCun initialisation), takes v=1/ninv = 1/n_{\text{in}}, so Var⁡(zj)=E[ai2]\operatorname{Var}(z_j) = \mathbb{E}[a_i^2]. If g(z)≈zg(z) \approx z near 0, as for tanh, that keeps the size of z\mathbf{z} about the same from layer to layer. The sigmoid does not fit that assumption: its slope at 0 is 1/4 and its outputs are centred on 1/2, not 0. Another common version uses v=2/(nin+nout)v = 2/(n_{\text{in}} + n_{\text{out}}); for the 8-to-8 hidden layers here the two agree.

    He initialisation is for ReLU. If zz is symmetric about 0, ReLU zeroes half of it and E[ReLU⁡(z)2]=12Var⁡(z)\mathbb{E}[\operatorname{ReLU}(z)^2] = \tfrac12\operatorname{Var}(z), so it takes v=2/ninv = 2/n_{\text{in}} to make up the half. A similar argument, with noutn_{\text{out}} in place of ninn_{\text{in}}, applies to the error signal on the way back. Both are aims about average sizes at the start, under these assumptions; neither promises anything once training moves the weights.

  5. Train both

    The deep sigmoid network with Xavier weights against the deep ReLU network with He weights. Press Train.

    Play steps through the lesson, and Train trains both networks, step for step. Both use plain gradient descent with η = 0.05 on the same batches of 20 points, for 100 epochs. The curves are the training loss after each epoch: binary cross-entropy, averaged over the 200 training points.

    The numbers here are for the end of the run. The ReLU network's loss falls from 1.257 to 0.209 by epoch 10 and to 0.0005 by epoch 100, with a few bumps on the way (the largest 0.260, at epoch 29). By epoch 100 every training point is classified correctly (100%).

    The sigmoid network makes no progress at telling the rings apart. Its loss does fall, from 0.710 to 0.693, but only for one reason. At the start its last hidden layer gives almost exactly the same values for every point: the output logit differs by at most 5.2×10⁻⁷ across the 200 training points, so every point gets the same probability of class 1, 0.409. The output layer's bias and weights can only shift that one shared prediction. Training moves it to 0.497 for every point after 100 epochs (to three decimals), close to the 1/2 that suits two classes of 100 points each, and that alone brings the loss down to 0.693, which is ln 2, the loss of saying 50% for every point. It has learnt nothing about where the points are.

    This page uses plain gradient descent on purpose. Adam divides each weight's step by the size of its own recent gradients, which would hide the gradient sizes this lesson is about.

    The maths

    Each step is W(ℓ)←W(ℓ)−η ∂LB/∂W(ℓ)\mathbf{W}^{(\ell)} \leftarrow \mathbf{W}^{(\ell)} - \eta\,\partial\mathcal{L}_B/\partial\mathbf{W}^{(\ell)} for every layer, and the same for the biases, where LB\mathcal{L}_B is the mean loss over a batch of 20. With one η for every layer, a layer whose gradient is 10−810^{-8} times another's takes a step 10−810^{-8} times as long. The loss ln⁡2\ln 2 is −ln⁡12-\ln\tfrac12, the log-loss of predicting probability 1/2 for every point.

  6. After the run, and further

    How far each layer's weights moved in those 100 epochs, compared with where they started.

    The bars are computed when you first reach this step, which takes a moment. In the sigmoid network, layer 1's weights changed by 1.7×10⁻⁷ of their starting size and layer 11's by 0.18: the early layers barely learnt. In the ReLU network every layer moved, layer 1 by 0.19 and layer 11 by 0.63.

    Two ideas make much deeper networks trainable, each named here in one line. Residual connections add a block's input to its output, a(ℓ+1)=a(ℓ)+f(a(ℓ))\mathbf{a}^{(\ell+1)} = \mathbf{a}^{(\ell)} + f(\mathbf{a}^{(\ell)}), so the gradient has a path back that skips the multiplications. Normalisation layers (batch normalisation, layer normalisation) rescale each layer's values during training, so they stay in a steady range as the weights change.

    The maths

    Each bar is ∥Wend(ℓ)−Wstart(ℓ)∥ / ∥Wstart(ℓ)∥\lVert\mathbf{W}^{(\ell)}_{\text{end}} - \mathbf{W}^{(\ell)}_{\text{start}}\rVert \,/\, \lVert\mathbf{W}^{(\ell)}_{\text{start}}\rVert, with Frobenius norms, after the same 100-epoch runs as step 5, on a log scale.

Quick check

Why do the first layers of this lesson's deep sigmoid network learn slowly?

Right. The error signal at layer ℓ is a product of one weight matrix and one set of slopes for every layer above it. On this page the gradient fell by a factor of about 5.1 per layer.

Layer 1 has 16 weights and layer 11 only 8, yet layer 11's gradient was about 11 million times larger. The number of weights is not the cause.

Plain gradient descent uses the same η for every layer. The early layers move less because their gradients are smaller.

The data enter at layer 1, and the forward pass reaches every layer. It is the error signal on the way back that shrinks.

What does He initialisation try to keep steady?

Right. With variance 2/n_in for the weights, and ReLU zeroing about half of each layer's values, the variance of z stays about the same from layer to layer, under the stated assumptions. On the way back the same factor keeps the error signal steady only when a layer has as many outputs as inputs, which holds for the 8-to-8 hidden layers here. Here every He bar was between 0.1 and 10.

Initialisation only sets the starting weights. It says nothing about how the loss moves once training starts.

η is chosen separately; He initialisation only scales the starting weights.

Training changes the weights; in the ReLU run every layer's weights moved. He initialisation only picks where they start.

Lesson complete

  • Backpropagation multiplies the error signal by a weight matrix and a set of slopes at every layer, so in a deep network the gradient can shrink or grow layer by layer.
  • The sigmoid's slope is at most 1/4; in this ten-layer sigmoid network with Xavier weights, layer 1's gradient started about 11 million times smaller than layer 11's, and training made no progress.
  • ReLU with He starting weights kept every layer's gradient between 0.1 and 10 here, and the same network trained; starting weights that were too small or too large broke it again.

Common misconceptions

  • "Gradients only vanish when the sigmoid units are saturated." Here they were near their steepest, with mean slopes between 0.22 and 0.24, and the gradient still fell by a factor of about 5.1 per layer. At best the slope is 1/4.
  • "Switching to ReLU fixes it." The starting scale matters as much: with ReLU and sd 0.1 every bar was below 10⁻⁶, and with sd 2 the largest was 2.0×10⁶.
  • "A deep sigmoid network can never be trained." This page shows one network, one start and plain gradient descent for 100 epochs. With sd 2 starting weights the sigmoid's ratio was 5.0×10⁻³ rather than 8.7×10⁻⁸; the starting weights, the optimiser, normalisation and residual connections all change the picture.
Next lesson: Convolutions →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 6, "Deep Feedforward Networks", on hidden units and back-propagation.
  2. Goodfellow, Bengio & Courville, Deep Learning, Chapter 8, "Optimization for Training Deep Models", on starting weights and normalisation.

About the data. The 200 training points are made up in your browser by the mulberry32 random number generator, seed 181, by the same rule as in the hidden-layers lesson. Each point gets an angle θ drawn uniformly from 0 to 2π and a radius r = r₀ + 0.25 z, where z is a draw from a normal distribution with mean 0 and SD 1; the point is (r cos θ, r sin θ). The first 100 points are class 0 with r₀ = 1 and the last 100 are class 1 with r₀ = 2.2. The starting weights use seed 182: every weight is a normal draw times a scale, the same draws for every setting. Xavier uses variance 1/nin, He 2/nin, and "sd 0.1" and "sd 2" use those standard deviations, for every layer including the output; biases start at 0. The batches of 20 are shuffled each epoch with seed 183.

If this helped, you can .