ML Explorer · Level 4 · Deep learning
Training a network
The craft of running training: small batches, passes through the data, and how to notice when a network starts memorising.
Overview
In hidden layers you pressed Train and watched a network bend its boundary, and in backpropagation you saw how each step finds its gradient. This lesson is about running those steps well.
Training usually does not use every point for every step. It works through the data in small batches, many steps per pass, and it runs for many passes. Run long enough, a big network can fit its training points so closely that it gets worse on new ones, so you watch the loss on points it was not trained on, and stop early or penalise large weights.
All the data are two interleaved spirals, made up in your browser. See "About the data" at the end.
You'll learn
- what a mini-batch and an epoch are, and why small batches give noisy loss curves
- how a big network can memorise its training points while its test loss turns up
- two fixes: stopping early, and weight decay
-
Batches
A training step does not need every point. It can work out the gradient on a small batch of points and step on that alone.
The chart shows three runs of the same network on the same 160 spiral points, each for 1600 examples seen, which is 10 passes through the data: in batches of 1 point, of 8, and of all 160 (the full batch). Each curve is the loss on the batch a step used, measured just before that step. Pick a batch size to bring its curve forward. The three runs are computed when you first reach this step, which takes a moment.
From one step to the next the batch loss changes by 0.284 on average with batch size 1, 0.0988 with 8 and 0.00679 with the full batch. One point says little about the loss on all 160, so its curve is noisy; the full batch's curve is smooth. But in the same 1600 examples, batch size 1 takes 1600 steps and the full batch only 10.
After the 1600 examples the loss on all 160 points is 0.661 for batch size 1, 0.650 for 8 and 0.667 for the full batch: on this run the smoothest curve did not end lowest.
Every run on this page uses the Adam optimiser with learning rate η = 0.01. Adam is a refinement of gradient descent, explained in the next lesson; the ideas in this lesson apply to plain gradient descent too.
The maths
Write for the loss on point , where holds every weight and bias. The full gradient and the gradient on a batch of points are
Each epoch the points are shuffled and cut into batches of equal size (1, 8, 16 and 160 all divide 160). At fixed weights, the batch gradients of one epoch average to exactly the full gradient, because together the batches use every point once: on average across the epoch's batches it equals the full gradient. Each batch is also a random sample of the points, drawn without replacement, so its gradient is an unbiased estimate of the full gradient. Coordinate by coordinate its variance is
where is the variance of that coordinate of the per-point gradients over the points (dividing by ). It shrinks as the batch grows and is 0 for the full batch. During training the weights move between batches, so each batch gradient is an estimate of the full gradient at the weights it is used at.
-
Epochs
One pass through all the training points is an epoch. With 160 points in batches of 16, an epoch is 160 / 16 = 10 steps.
Press Train. Play steps through the lesson, and Train trains the network. On this step training runs slowly: the 16 points ringed in white are the batch the last step used, and the read-out counts the steps, the finished epochs, and the batches done in the current epoch. Moving to another step pauses training.
At the start of each epoch the points are shuffled into a new order, so the batches change from one epoch to the next, but within an epoch every point is used exactly once. The shading is the network's , its probability of class 1, with the boundary drawn in white.
The maths
With training points and batch size , one epoch is steps, so training for epochs takes steps. When does not divide , the last batch of each epoch is smaller; every batch size on this page divides its training set, so that never happens here.
-
Press Train
Now at full speed. The network has two hidden layers of 16 tanh units; watch its boundary wind along the spiral.
This is step 2's run, continued: batches of 16, η = 0.01, for 300 epochs. The curve top right is the training loss after each epoch: binary cross-entropy, the log-loss of logistic regression, averaged over the training points.
On this seeded run (the end-of-run numbers here are for the full 300-epoch run; the read-out says finished once the run is done) the training loss falls from 0.72 to 0.064, and the network gets 98.1% of the 160 training points right and 91.3% of the 400 test points.
"Learns" here means adjusts its weights to lower the loss on the training data. The test points show how it does on points it never trained on.
The maths
The network, with layers numbered as on the notation page:
σ here is the sigmoid, not a standard deviation. With 16 units in each hidden layer it has 337 weights and biases. ℒ is the mean of the per-point log-loss over the training points; the gradient of each batch's mean is found by backpropagation, and Adam turns it into a step.
-
Memorising
Give a much bigger network fewer points and it can fit them too closely: the training loss keeps falling while the test loss turns up.
This network has two hidden layers of 64 units, 4417 weights and biases, and only 60 training points, a new spiral sample. It trains in batches of 20, so an epoch is 3 steps, for 400 epochs. The chart shows the training loss (cyan, solid) and the loss on the 400 test points (violet, dashed) after every epoch; the dotted line marks the epoch with the lowest test loss so far.
On this seeded run (the end-of-run numbers here are for the full 400-epoch run; the read-out says finished once the run is done) the test loss is lowest at epoch 120, at 0.474. After that the training loss falls overall, from 0.139 to 0.004, but the test loss climbs overall to 0.584 by the end, 23% above its best. The network ends with 100% of the training points right and 81.5% of the test points.
This is the overfitting picture again: the network keeps fitting its 60 points more closely, and on this run that extra fit makes it worse on new points. One fix is early stopping: keep the weights from the epoch with the lowest loss on held-back points, here epoch 120, rather than the last.
Strictly, that choice should be made on a separate validation set, keeping the test set for the final check, as in train and test. Here the test points stand in, to keep the picture simple.
The maths
Early stopping records the validation loss after each epoch and returns the weights from
In practice training stops once the validation loss has not improved for a set number of epochs, the patience, and the best weights seen are kept. Stopping early also keeps the weights nearer their small starting values; for a simple quadratic loss this can be shown to act much like weight decay, the next step.
-
Weight decay
Another fix: add a penalty for large weights to the loss, as ridge regression did. Each step then also pulls every weight a little towards 0.
Switch weight decay on (λ = 0.001) and press Train. Everything else is as in step 4: the same starting weights and the same batches. Once both settings have been trained a little, the other setting's test curve stays on the chart, dotted grey.
The curves show the data loss, the log-loss without the penalty, so the two settings are compared on the same footing.
On this seeded run (the end-of-run numbers here are for the full 400-epoch run; the read-out says finished once the run is done) the final test loss is 0.359 with weight decay against 0.584 without. With weight decay the test loss does not turn up: it is still falling overall at the end, with its lowest value, 0.341, at epoch 399. Without it, the test loss was lowest at epoch 120, at 0.474, and then climbed overall to 0.584. The curve with weight decay still wobbles from epoch to epoch. The training loss ends higher, 0.031 against 0.004: the penalty gives up a little fit to the training points. Test accuracy at the end is 87.0% against 81.5%.
λ = 0.001 was picked by trying a few values on these runs. In practice λ is chosen on a validation set, like ridge's λ in the overfitting lesson.
The maths
Weight decay trains on the penalised loss
where is the sum of the squares of every weight in the matrix; the biases are not penalised. The penalty's gradient is , so each weight's gradient gains . With plain gradient descent the step becomes
so every weight shrinks by the factor before the usual step, hence the name. λ plays the same role as ridge's λ; there the penalty was written without the ½, which only rescales λ. With Adam, as on this page, is added to the gradient before Adam rescales the step, so the shrink is not exactly that factor; a variant called AdamW applies the shrink directly.
-
The knobs
Four numbers set how training runs. None has one right value; you watch the loss curves and choose. The table gives the values used in steps 2–3 and steps 4–5.
Batch size: smaller batches give noisier steps but more of them per epoch (step 1). Epochs: too few and training has not finished; too many and a big network can memorise (step 4). η: how far each step goes, as in gradient descent. λ: how hard large weights are penalised (step 5).
The last column is general guidance, not something this page showed: this page never used a learning rate that was too large or a λ that was too strong. A learning rate that is too large is shown in gradient descent.
What to watch: the loss on points the network was not trained on. On this page that was the test set; properly it is a validation set held back for these choices, with the test set kept for one final check, as in train and test.
Quick check
Lesson complete
- Training takes many small steps, each on a batch; an epoch is one pass through all the training points.
- Smaller batches give noisier loss curves but more steps per epoch; each batch gradient estimates the full gradient.
- A big network can memorise its training points: watch the loss on held-back points, and stop early or add weight decay.
Common misconceptions
- "A lower training loss means a better model." In step 4, after epoch 120 the training loss fell from 0.139 to 0.004 while the test loss rose from 0.474 to 0.584.
- "The smoothest loss curve trains best." In step 1 the full batch gave the smoothest curve, but after the same 1600 examples it had taken 10 steps and its training loss, 0.667, was not lower than batch size 8's 0.650.
- "Early stopping means stopping the first time the test loss goes up." In step 4 the test loss went up from one epoch to the next 45 times before reaching its lowest point at epoch 120. In practice you wait a set number of epochs and keep the best weights seen.
Further reading
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 8, Section 8.1.3, "Batch and Minibatch Algorithms", on training with gradients from small batches.
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 7, Section 7.1.1, "L2 Parameter Regularization" (weight decay), and Section 7.8, "Early Stopping".
- Bishop, Pattern Recognition and Machine Learning, its chapter on neural networks, which covers regularisation and early stopping.
About the data. All three sets are two interleaved spirals, made up in your browser by the mulberry32 random number generator. A set of n points has n/2 on each arm: point i of arm c (c = 0 or 1, i = 0, 1, …, n/2 − 1) has t = i/(n/2), radius 0.3 + 2.7t and angle 3πt + πc, so each arm makes one and a half turns; it sits at (radius × cos angle, radius × sin angle) plus 0.2 z on each coordinate, where z is a draw from a normal distribution with mean 0 and SD 1. Its class is its arm, c. The 160 training points use seed 171, the 400 test points seed 172, and step 4's 60 training points seed 175. Both networks start from weights drawn with seed 173, each a normal draw with mean 0 and variance 1/(number of inputs to that layer), with biases 0. Each epoch's shuffle of the training points comes from seed 174. Every run uses Adam with η = 0.01 and its usual settings (β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸); the weight-decay run differs from step 4's only in λ.
If this helped, you can .