ML Explorer · Level 4 · Deep learning
Going deeper: vanishing gradients
Stack ten hidden layers of sigmoid units and the gradient that reaches the first layer can be millions of times smaller than at the last. Here you measure it layer by layer, see why, and watch a ReLU network with better starting weights train where the sigmoid one stalls.
Overview
The networks so far had one hidden layer. Adding more lets a network build features out of features, but it makes training harder. Backpropagation passes the error signal backwards one layer at a time, multiplying it by something at every layer. Over many layers those factors pile up.
This lesson takes a network with ten hidden layers, measures the size of the gradient for each layer's weights at the start of training, and shows how the choice of activation and of starting weights changes it. Then it trains two deep networks side by side.
The data are two rings like those in hidden layers, made up in your browser; see "About the data" at the end.
You'll learn
- why, in a deep sigmoid network, the gradient for the first layers can be far smaller than for the last
- how ReLU with He starting weights keeps the gradient about the same size from layer to layer here
- what Xavier and He initialisation aim for, and what they assume
-
Stack more layers
Two inputs, ten hidden layers of 8 units, one output: 681 weights and biases.
Each hidden layer takes the 8 values of the layer before, forms 8 weighted sums plus biases, and passes each through the same activation g. There are 11 layers of weights, numbered ℓ = 1 (out of the inputs) to ℓ = 11 (into the output). In steps 2 and 3 every hidden unit is a sigmoid. σ here is the sigmoid, not a standard deviation.
One hidden layer would be plenty for these rings. The extra depth is here to show what depth does to training, not to win accuracy.
The maths
For layers with and :
for the hidden layers . The output is a single number with no activation; its sigmoid is the predicted probability of class 1. The count is .
-
Gradient size per layer, at the start
One bar per layer of weights: how big the gradient for that layer is before any training. The chart uses a log scale: each grid line is 100 times the one below.
With sigmoid units and Xavier starting weights, the gradient for layer 11 has size 0.14. Each layer back it is smaller, and at layer 1 it is 1.2×10⁻⁸: 8.7×10⁻⁸ times layer 11's, or about 11 million times smaller.
Plain gradient descent moves each weight by η times its gradient, so with one η for the whole network the first layers barely move while the last ones do. On the log scale the bars climb like a staircase from left to right.
The maths
ℒ is the mean loss over all 200 training points, each point's loss being binary cross-entropy, the log-loss of logistic regression. Each bar is the Frobenius norm of the gradient for one weight matrix, the square root of the sum of its squared entries:
computed by backpropagation at the starting weights. The biases are left out of the bars.
-
Why it shrinks
On the way back, every sigmoid layer multiplies the error signal by its slope σ′, and that slope is never more than 1/4.
Backpropagation gets layer ℓ's error signal from layer ℓ + 1's: multiply by layer ℓ + 1's weights, transposed, then by the slope of the activation at each unit of layer ℓ. For the sigmoid that slope is at most 1/4, reached only at z = 0 (the inset).
Xavier starting weights are scaled so that, on average, multiplying by the weights keeps the signal about the same size. So the slopes decide. At the start the sigmoid units sit near their steepest: averaged over the training points, the slope in each hidden layer is between 0.22 and 0.24. Even so, each factor is under a quarter, and on average the gradient size falls by a factor of 5.1 per layer from layer 11 back to layer 1.
That is this network with these starting weights. Whether the gradient shrinks, holds or grows depends on the activation and on the size of the weights, which is what the next step lets you change.
The maths
Here, for one training point, ℒ is that point's loss (the bars average the resulting gradients over the 200 points). With , backpropagation gives
where ⊙ multiplies entry by entry, starting from . Unrolled, the error signal at layer ℓ is a product of one weight matrix and one set of slopes for every layer above it:
For the sigmoid, , with equality exactly when , that is . So each step back obeys
where is the most that can stretch a vector (the same for ). Across ten sigmoid layers the slopes alone contribute at most ; unless the weights' stretches multiply to more than , the bound shrinks. It is a bound, not the actual size, which is why the page measures the bars instead. The step's factor of 5.1 per layer is the tenth root of the measured ratio, , an average over ten steps back; the steps differ, and the largest is the last one, from layer 2 to layer 1, a factor of about 22. For tanh the slope is at most 1, and for ReLU it is 0 or 1.
-
Activation and starting weights
Pick the activation and the size of the starting weights; the bars are recomputed for the same network and data.
"Xavier / He" means Xavier for sigmoid and tanh and He for ReLU. "sd 0.1" and "sd 2" draw every weight with that standard deviation, whatever the layer. All settings use the same random draws, only scaled, and every bias starts at 0.
ReLU with He: every bar is between 0.1 and 10, and layer 1's is 0.47 times layer 11's. Tanh with Xavier holds steady too, with a ratio of 0.42; tanh's slope near 0 is 1, not 1/4.
Too small, sd 0.1: with ReLU or tanh every bar is below 10⁻⁶. Small weights shrink the signal on the way forward and the error signal on the way back, so every layer gets almost nothing. With the sigmoid, layer 1's bar drops below 10⁻¹².
Too large, sd 2: with ReLU every bar shoots past the top of the chart; the largest is 2.0×10⁶, against 3.3 with He. Bars that pass 10² are cut off and marked with a triangle. With tanh, layer 1's gradient is 320 times layer 11's. With the sigmoid, sd 2 helps: the ratio is 5.0×10⁻³, still under a hundredth.
The maths
Take one unit, , and assume the weights are independent, with mean 0 and variance , and independent of the inputs , which are themselves independent of each other with the same distribution. Then
Xavier initialisation, in the version used here (also called LeCun initialisation), takes , so . If near 0, as for tanh, that keeps the size of about the same from layer to layer. The sigmoid does not fit that assumption: its slope at 0 is 1/4 and its outputs are centred on 1/2, not 0. Another common version uses ; for the 8-to-8 hidden layers here the two agree.
He initialisation is for ReLU. If is symmetric about 0, ReLU zeroes half of it and , so it takes to make up the half. A similar argument, with in place of , applies to the error signal on the way back. Both are aims about average sizes at the start, under these assumptions; neither promises anything once training moves the weights.
-
Train both
The deep sigmoid network with Xavier weights against the deep ReLU network with He weights. Press Train.
Play steps through the lesson, and Train trains both networks, step for step. Both use plain gradient descent with η = 0.05 on the same batches of 20 points, for 100 epochs. The curves are the training loss after each epoch: binary cross-entropy, averaged over the 200 training points.
The numbers here are for the end of the run. The ReLU network's loss falls from 1.257 to 0.209 by epoch 10 and to 0.0005 by epoch 100, with a few bumps on the way (the largest 0.260, at epoch 29). By epoch 100 every training point is classified correctly (100%).
The sigmoid network makes no progress at telling the rings apart. Its loss does fall, from 0.710 to 0.693, but only for one reason. At the start its last hidden layer gives almost exactly the same values for every point: the output logit differs by at most 5.2×10⁻⁷ across the 200 training points, so every point gets the same probability of class 1, 0.409. The output layer's bias and weights can only shift that one shared prediction. Training moves it to 0.497 for every point after 100 epochs (to three decimals), close to the 1/2 that suits two classes of 100 points each, and that alone brings the loss down to 0.693, which is ln 2, the loss of saying 50% for every point. It has learnt nothing about where the points are.
This page uses plain gradient descent on purpose. Adam divides each weight's step by the size of its own recent gradients, which would hide the gradient sizes this lesson is about.
The maths
Each step is for every layer, and the same for the biases, where is the mean loss over a batch of 20. With one η for every layer, a layer whose gradient is times another's takes a step times as long. The loss is , the log-loss of predicting probability 1/2 for every point.
-
After the run, and further
How far each layer's weights moved in those 100 epochs, compared with where they started.
The bars are computed when you first reach this step, which takes a moment. In the sigmoid network, layer 1's weights changed by 1.7×10⁻⁷ of their starting size and layer 11's by 0.18: the early layers barely learnt. In the ReLU network every layer moved, layer 1 by 0.19 and layer 11 by 0.63.
Two ideas make much deeper networks trainable, each named here in one line. Residual connections add a block's input to its output, , so the gradient has a path back that skips the multiplications. Normalisation layers (batch normalisation, layer normalisation) rescale each layer's values during training, so they stay in a steady range as the weights change.
The maths
Each bar is , with Frobenius norms, after the same 100-epoch runs as step 5, on a log scale.
Quick check
Lesson complete
- Backpropagation multiplies the error signal by a weight matrix and a set of slopes at every layer, so in a deep network the gradient can shrink or grow layer by layer.
- The sigmoid's slope is at most 1/4; in this ten-layer sigmoid network with Xavier weights, layer 1's gradient started about 11 million times smaller than layer 11's, and training made no progress.
- ReLU with He starting weights kept every layer's gradient between 0.1 and 10 here, and the same network trained; starting weights that were too small or too large broke it again.
Common misconceptions
- "Gradients only vanish when the sigmoid units are saturated." Here they were near their steepest, with mean slopes between 0.22 and 0.24, and the gradient still fell by a factor of about 5.1 per layer. At best the slope is 1/4.
- "Switching to ReLU fixes it." The starting scale matters as much: with ReLU and sd 0.1 every bar was below 10⁻⁶, and with sd 2 the largest was 2.0×10⁶.
- "A deep sigmoid network can never be trained." This page shows one network, one start and plain gradient descent for 100 epochs. With sd 2 starting weights the sigmoid's ratio was 5.0×10⁻³ rather than 8.7×10⁻⁸; the starting weights, the optimiser, normalisation and residual connections all change the picture.
Further reading
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 6, "Deep Feedforward Networks", on hidden units and back-propagation.
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 8, "Optimization for Training Deep Models", on starting weights and normalisation.
About the data. The 200 training points are made up in your browser by the mulberry32 random number generator, seed 181, by the same rule as in the hidden-layers lesson. Each point gets an angle θ drawn uniformly from 0 to 2π and a radius r = r₀ + 0.25 z, where z is a draw from a normal distribution with mean 0 and SD 1; the point is (r cos θ, r sin θ). The first 100 points are class 0 with r₀ = 1 and the last 100 are class 1 with r₀ = 2.2. The starting weights use seed 182: every weight is a normal draw times a scale, the same draws for every setting. Xavier uses variance 1/nin, He 2/nin, and "sd 0.1" and "sd 2" use those standard deviations, for every layer including the output; biases start at 0. The batches of 20 are shuffled each epoch with seed 183.
If this helped, you can .