ML Explorer · Level 4 · Deep learning
Recurrent networks
A recurrent network reads a sequence one step at a time and carries a hidden state forward. Train one live to repeat the bit it saw a few steps ago, and see why reaching further back is harder.
Overview
Every network so far took its whole input at once: two numbers, or an 8 × 8 image in convolutional networks. Much data arrives as a sequence instead: words, sounds, readings over time. A recurrent network (RNN) reads one item at a time and keeps a short running summary of what it has read, its hidden state, which it updates at every step.
The task here is made up, so the right answer is known exactly: a stream of random bits goes in, and at every step the network has to say the bit it saw k steps ago. You will see the loop drawn out step by step, train it live for k = 2, try k = 1, 2, 4 and 8, and look at why reaching further back is harder.
You'll learn
- how one cell, used again at every step, reads a sequence and carries a hidden state forward
- that training unrolls the loop and runs backpropagation through it, which is called backpropagation through time
- why the gradient from far back tends to shrink, and the idea behind gated cells such as the LSTM
-
A stream of bits
Each stream is 12 random bits, each 0 or 1 with equal chance. The task: at every step, say the bit from k steps ago.
With k = 2, the target at step 5 is the bit at step 3, the target at step 8 is the bit at step 6, and so on; the arrows mark one in each stream. The first 2 steps have no target (–), because nothing came 2 steps before them, so they are left out of the loss and out of the accuracy.
There are 200 training streams, and a separate 200 test streams made the same way with a different seed, used only to check the network, as in train and test. Each target bit is as likely to be 0 as 1, so guessing is right half the time: chance here is 50%.
The maths
A stream is with (here is the number of steps, not a transpose). The target is
and steps have none. So the 200 test streams hold steps with a target: 2,000 for and 800 for . The recipe is in "About the data" at the end.
-
The loop
One small cell reads the bit and the previous hidden state , and gives the new hidden state : 8 numbers between −1 and 1. The same cell, with the same weights, is used at every step.
Drawn as a loop it is one box with an arrow back into itself. Drawn unrolled, as here, it is a row of copies of that box, one per step, with an arrow carrying the hidden state along. Each column fills as the stream is read: the bit, the 8 numbers of (cyan for positive, violet with a minus sign for negative), the network's probability that the target is 1, and the target. Move the step t slider to read it one step at a time.
This is the starting network, with random weights, so its probabilities are guesses: at step 12 it gives = 0.67. It has 89 weights and biases in all, however long the stream is.
The maths
With , at each step :
with tanh applied to each of the 8 numbers, and σ here is the sigmoid, not a standard deviation. , and have 8 entries each, is 8 × 8 and is one number: . The predicted bit is 1 when (that is, ) and 0 otherwise, so counts as 0.
-
Train it (k = 2)
Press Train. The network is trained to repeat the bit from 2 steps ago, and the curve shows the masked test accuracy after every batch.
Play steps through the lesson, and Train trains the network. Training runs Adam (from optimisers) with η = 0.02 on batches of 10 streams for 10 epochs, which is 200 batches. The masked test accuracy is the share of the 2,000 test steps that have a target (steps 3 to 12 of each of the 200 test streams) where the predicted bit equals the target.
These numbers are for the whole run. At the start the masked test accuracy is 59.5% (a random start can sit a little above the 50% of chance on one test set) and the training loss ℒ is 0.700. After batch 6 it is 97.7%, up from 92.3% one batch before, and it stays at or above 95% from then on. From batch 18 every one of the 2,000 masked test steps is right, to the end: 100%. The loss falls every epoch, to 0.0018 after epoch 10.
The top row is test stream 1 with the current , each marked ✓ or ✗ against its target. The first epoch is shown slowly, one batch every few frames, so the rise can be watched; the numbers are the same at any speed. Reset goes back to the same starting weights, and the next run is the same as the last.
The maths
The loss for one stream is the binary cross-entropy, which is the log-loss of logistic regression, averaged over the steps that have a target:
A batch's loss is the mean of this over its 10 streams, and the read-out's training loss is the mean over all 200 training streams at the end of each epoch.
Backpropagation through time is backpropagation on the unrolled picture: a 12-layer network whose layers all share the same weights. Writing and for one stream's loss, the weight used at every step collects a gradient from every step:
Adam then moves each of the 89 weights and biases, with , , .
-
Further back
Move k from 1 to 8. Each k trains a new network from the same starting weights, with the same settings and the same budget: 10 epochs, 200 batches.
The bit streams are the same for every k; only the targets change. After 10 epochs, k = 1 gets 100% of its masked test steps right, k = 2 100% and k = 4 100%. k = 1 and 2 are there after the first epoch; k = 4 takes until epoch 4.
k = 8 starts at 50%, chance, and ends at 88.9%: 711 of its 800 masked test steps. It stays below 95% at every epoch of the run; its best epoch reached 88.9%. On its own training streams it gets 90.4%, so within this budget it has not fitted the training streams either.
Why is k = 8 harder? Not mainly because there is more to hold: in these 12-step streams the most bits the hidden state ever has to carry at once is 4 for k = 8, fewer than the 5 for k = 4, which reached 100%. The difference is distance. Each bit has to be carried through 8 updates of the cell before it is asked for ( is needed at step 9), and at the start the training signal that links a target to the bit 8 steps back is much weaker; the next step shows it. It also has fewer targets to learn from: 4 per stream, against 10 for k = 2. This is one data set and one starting network, and a longer run or another seed could give other numbers.
The maths
For k = 8 the only targets are , asked for at steps 9 to 12, so each has to survive 8 updates of . After reading , the hidden state must still carry every bit that is needed now or later: each with whose target step is at most 12. The most at any one step is 2 for k = 1, 3 for k = 2, 5 for k = 4 and 4 for k = 8. So k = 8 asks for fewer bits at once than k = 4, but over a longer distance.
Chance is 50% because each target bit is 0 or 1 with equal chance, whatever the network has seen most recently, so a guess that ignores the bit 8 steps back is right half the time on average.
The runs stop at 10 epochs because beyond about epoch 17 the k = 8 run becomes sensitive to tiny rounding differences: a change in the twelfth digit of the starting weights changes its later numbers, so numbers past that point could differ between browsers.
-
Why far back is hard
How strongly does the loss at the last step respond to each earlier hidden state? The plot shows the size of that gradient, on a log scale, for the starting network.
Each thin dashed line is one test stream; the thick line is the typical size over all 200 (the geometric mean, which is the average on the log scale). At the last step it is 0.37; 11 steps back it is 0.012, about 30 times smaller. 2 steps back it is about 3 times smaller than at the last step, and 8 steps back about 16 times smaller.
So at the start, the signal that ties a target to the bit 8 steps earlier is much weaker than the one for 2 steps, and it arrives mixed with the signals from the more recent bits, which carry nothing about that target. This is the vanishing gradient of deep networks again: unrolled, the network is 12 layers deep.
It is a tendency, not a rule. Training changes : after step 4's k = 8 run, measured on the k = 8 test targets, the typical size is 0.37 at the last step and 0.94 11 steps back, larger rather than smaller, yet that network still ended at 88.9%. A weak signal at the start is part of why far back is hard, not the whole story.
The maths
Here is the cross-entropy of the last step alone, and the plot shows against . By the chain rule,
and is a product of Jacobians, one per step:
where is the slope of tanh at each unit, between 0 and 1, and multiplies entry by entry. Each factor can shrink a vector or stretch it. If every factor shrinks vectors by at least some fixed fraction, the product shrinks geometrically with the distance ; if they stretch, it can grow. Since the slopes are at most 1, , where is the largest factor by which stretches a vector. For the starting network it is 1.82, so this bound allows growth; the shrinking in the plot comes from the actual products, not from the bound.
The target only multiplies each line by , so the shape of each line does not depend on k. The plot uses the k = 2 targets.
-
Gates
A plain RNN passes everything through and tanh at every step. Gated cells, such as the LSTM and the GRU, add a path that can carry information forward unchanged.
An LSTM (long short-term memory) cell keeps a second state, the cell state , alongside . From one step to the next, the cell state is only multiplied by a forget gate and added to through an input gate. Each gate is a set of numbers between 0 and 1, computed from and with a sigmoid and its own weights. With the forget gate near 1 and the input gate near 0, the cell state passes on unchanged unless a gate says otherwise, and along that path the gradient is not squeezed by and tanh at each step. Training sets the gates' weights, so the network adjusts when to keep, add and forget. The GRU (gated recurrent unit) uses the same idea with two gates and no separate cell state. Gates make long memory easier to learn; they do not make it free, and neither cell is trained on this page.
Another way is to stop squeezing the whole past into one hidden state and let each step look straight back at every earlier step, choosing what to use. That idea is attention, and Level 5 builds up to it.
The maths
The core of the LSTM is the cell-state update
where , and are the forget, input and output gates and holds new candidate values, each computed from and . Along the cell-state path alone, the factor from one step to the next is , with no and no tanh slope; the full gradient also has paths through the gates and .
Quick check
Lesson complete
- An RNN uses one cell at every step, , and the hidden state carries a summary of the stream forward.
- Training is backpropagation on the unrolled loop. Within 10 epochs this network got every k = 2 test step right but reached only 88.9% at k = 8.
- The gradient from far back passes through a product of Jacobians and, for the starting network here, shrinks with distance; gated cells such as the LSTM and GRU add a path that can carry information forward unchanged.
Common misconceptions
- "An RNN has separate weights for each step." The same 89 weights and biases are used at every step, which is why each weight's gradient is a sum over steps.
- "The hidden state stores the whole past." It is 8 numbers; at k = 8 this network got only 88.9% of the masked test steps right.
- "The gradient through time always vanishes." For the starting network it shrank about 30 times over 11 steps, but after the k = 8 run it was larger 11 steps back (0.94) than at the last step.
- "Gates remove the problem of long memory." They give information and gradients an easier path forward and back, which makes long memory easier to learn, not guaranteed.
That is the end of Level 4. Level 5 starts with how text becomes numbers.
Next lesson: Tokens: text as numbers →Further reading
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 10, "Sequence Modeling: Recurrent and Recursive Nets", on recurrent networks, backpropagation through time, long-term dependencies and gated cells such as the LSTM and GRU.
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), on recurrent neural networks.
About the data. Every stream is 12 bits made up in your browser by the mulberry32 random number generator: for each stream in turn, 12 uniform draws u, and the bit is 1 if u < 0.5 and 0 otherwise. The target at step t is the bit at step t − k, for t = k + 1 to 12; the first k steps have none. The 200 training streams use seed 211 and the 200 test streams seed 212, for every k, so the streams are the same for every k and only the targets change. The network's starting weights use seed 213, in this order: the 8 input weights, each a normal draw with mean 0 and SD 1; the 64 entries of the 8 × 8 recurrent matrix, row by row, SD 1/√8; the 8 output weights, SD 1/√8; the biases start at 0. Each epoch's shuffle of the training streams comes from seed 214. Every run uses Adam with η = 0.02, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, batches of 10 streams and 10 epochs.
If this helped, you can .