Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Training a network

The craft of running training: small batches, passes through the data, and how to notice when a network starts memorising.

About 11 minutes · Live training on the stage; the formulas are in The maths

Overview

In hidden layers you pressed Train and watched a network bend its boundary, and in backpropagation you saw how each step finds its gradient. This lesson is about running those steps well.

Training usually does not use every point for every step. It works through the data in small batches, many steps per pass, and it runs for many passes. Run long enough, a big network can fit its training points so closely that it gets worse on new ones, so you watch the loss on points it was not trained on, and stop early or penalise large weights.

All the data are two interleaved spirals, made up in your browser. See "About the data" at the end.

You'll learn

  • what a mini-batch and an epoch are, and why small batches give noisy loss curves
  • how a big network can memorise its training points while its test loss turns up
  • two fixes: stopping early, and weight decay
  1. Batches

    A training step does not need every point. It can work out the gradient on a small batch of points and step on that alone.

    The chart shows three runs of the same network on the same 160 spiral points, each for 1600 examples seen, which is 10 passes through the data: in batches of 1 point, of 8, and of all 160 (the full batch). Each curve is the loss on the batch a step used, measured just before that step. Pick a batch size to bring its curve forward. The three runs are computed when you first reach this step, which takes a moment.

    From one step to the next the batch loss changes by 0.284 on average with batch size 1, 0.0988 with 8 and 0.00679 with the full batch. One point says little about the loss on all 160, so its curve is noisy; the full batch's curve is smooth. But in the same 1600 examples, batch size 1 takes 1600 steps and the full batch only 10.

    After the 1600 examples the loss on all 160 points is 0.661 for batch size 1, 0.650 for 8 and 0.667 for the full batch: on this run the smoothest curve did not end lowest.

    Every run on this page uses the Adam optimiser with learning rate η = 0.01. Adam is a refinement of gradient descent, explained in the next lesson; the ideas in this lesson apply to plain gradient descent too.

    The maths

    Write Li(θ)\mathcal{L}_i(\boldsymbol{\theta}) for the loss on point ii, where θ\boldsymbol{\theta} holds every weight and bias. The full gradient and the gradient on a batch BB of nBn_B points are

    ∇L(θ)=1n∑i=1n∇Li(θ),gB(θ)=1nB∑i∈B∇Li(θ)\nabla\mathcal{L}(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} \nabla\mathcal{L}_i(\boldsymbol{\theta}), \qquad \mathbf{g}_B(\boldsymbol{\theta}) = \frac{1}{n_B}\sum_{i \in B} \nabla\mathcal{L}_i(\boldsymbol{\theta})

    Each epoch the points are shuffled and cut into n/nBn/n_B batches of equal size (1, 8, 16 and 160 all divide 160). At fixed weights, the batch gradients of one epoch average to exactly the full gradient, because together the batches use every point once: on average across the epoch's batches it equals the full gradient. Each batch is also a random sample of the points, drawn without replacement, so its gradient is an unbiased estimate of the full gradient. Coordinate by coordinate its variance is

    Var⁡(gB)=s2nB⋅n−nBn−1\operatorname{Var}\big(\mathbf{g}_B\big) = \frac{s^2}{n_B}\cdot\frac{n - n_B}{n - 1}

    where s2s^2 is the variance of that coordinate of the per-point gradients over the nn points (dividing by nn). It shrinks as the batch grows and is 0 for the full batch. During training the weights move between batches, so each batch gradient is an estimate of the full gradient at the weights it is used at.

  2. Epochs

    One pass through all the training points is an epoch. With 160 points in batches of 16, an epoch is 160 / 16 = 10 steps.

    Press Train. Play steps through the lesson, and Train trains the network. On this step training runs slowly: the 16 points ringed in white are the batch the last step used, and the read-out counts the steps, the finished epochs, and the batches done in the current epoch. Moving to another step pauses training.

    At the start of each epoch the points are shuffled into a new order, so the batches change from one epoch to the next, but within an epoch every point is used exactly once. The shading is the network's p^\hat{p}, its probability of class 1, with the boundary p^=0.5\hat{p} = 0.5 drawn in white.

    The maths

    With nn training points and batch size nBn_B, one epoch is n/nBn/n_B steps, so training for EE epochs takes E⋅n/nBE \cdot n/n_B steps. When nBn_B does not divide nn, the last batch of each epoch is smaller; every batch size on this page divides its training set, so that never happens here.

  3. Press Train

    Now at full speed. The network has two hidden layers of 16 tanh units; watch its boundary wind along the spiral.

    This is step 2's run, continued: batches of 16, η = 0.01, for 300 epochs. The curve top right is the training loss after each epoch: binary cross-entropy, the log-loss of logistic regression, averaged over the training points.

    On this seeded run (the end-of-run numbers here are for the full 300-epoch run; the read-out says finished once the run is done) the training loss falls from 0.72 to 0.064, and the network gets 98.1% of the 160 training points right and 91.3% of the 400 test points.

    "Learns" here means adjusts its weights to lower the loss on the training data. The test points show how it does on points it never trained on.

    The maths

    The network, with layers numbered as on the notation page:

    a(1)=tanh⁡(W(1)x+b(1)),a(2)=tanh⁡(W(2)a(1)+b(2)),p^=σ(W(3)a(2)+b(3))\mathbf{a}^{(1)} = \tanh\big(\mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}\big), \quad \mathbf{a}^{(2)} = \tanh\big(\mathbf{W}^{(2)}\mathbf{a}^{(1)} + \mathbf{b}^{(2)}\big), \quad \hat{p} = \sigma\big(\mathbf{W}^{(3)}\mathbf{a}^{(2)} + b^{(3)}\big)

    σ here is the sigmoid, not a standard deviation. With 16 units in each hidden layer it has 337 weights and biases. ℒ is the mean of the per-point log-loss over the training points; the gradient of each batch's mean is found by backpropagation, and Adam turns it into a step.

  4. Memorising

    Give a much bigger network fewer points and it can fit them too closely: the training loss keeps falling while the test loss turns up.

    This network has two hidden layers of 64 units, 4417 weights and biases, and only 60 training points, a new spiral sample. It trains in batches of 20, so an epoch is 3 steps, for 400 epochs. The chart shows the training loss (cyan, solid) and the loss on the 400 test points (violet, dashed) after every epoch; the dotted line marks the epoch with the lowest test loss so far.

    On this seeded run (the end-of-run numbers here are for the full 400-epoch run; the read-out says finished once the run is done) the test loss is lowest at epoch 120, at 0.474. After that the training loss falls overall, from 0.139 to 0.004, but the test loss climbs overall to 0.584 by the end, 23% above its best. The network ends with 100% of the training points right and 81.5% of the test points.

    This is the overfitting picture again: the network keeps fitting its 60 points more closely, and on this run that extra fit makes it worse on new points. One fix is early stopping: keep the weights from the epoch with the lowest loss on held-back points, here epoch 120, rather than the last.

    Strictly, that choice should be made on a separate validation set, keeping the test set for the final check, as in train and test. Here the test points stand in, to keep the picture simple.

    The maths

    Early stopping records the validation loss after each epoch ee and returns the weights from

    e^=arg⁡min⁡e Lval(e)\hat{e} = \arg\min_{e}\ \mathcal{L}_{\text{val}}(e)

    In practice training stops once the validation loss has not improved for a set number of epochs, the patience, and the best weights seen are kept. Stopping early also keeps the weights nearer their small starting values; for a simple quadratic loss this can be shown to act much like weight decay, the next step.

  5. Weight decay

    Another fix: add a penalty for large weights to the loss, as ridge regression did. Each step then also pulls every weight a little towards 0.

    Switch weight decay on (λ = 0.001) and press Train. Everything else is as in step 4: the same starting weights and the same batches. Once both settings have been trained a little, the other setting's test curve stays on the chart, dotted grey.

    The curves show the data loss, the log-loss without the penalty, so the two settings are compared on the same footing.

    On this seeded run (the end-of-run numbers here are for the full 400-epoch run; the read-out says finished once the run is done) the final test loss is 0.359 with weight decay against 0.584 without. With weight decay the test loss does not turn up: it is still falling overall at the end, with its lowest value, 0.341, at epoch 399. Without it, the test loss was lowest at epoch 120, at 0.474, and then climbed overall to 0.584. The curve with weight decay still wobbles from epoch to epoch. The training loss ends higher, 0.031 against 0.004: the penalty gives up a little fit to the training points. Test accuracy at the end is 87.0% against 81.5%.

    λ = 0.001 was picked by trying a few values on these runs. In practice λ is chosen on a validation set, like ridge's λ in the overfitting lesson.

    The maths

    Weight decay trains on the penalised loss

    J(θ)=L(θ)+λ2∑ℓ∥W(ℓ)∥2J(\boldsymbol{\theta}) = \mathcal{L}(\boldsymbol{\theta}) + \frac{\lambda}{2}\sum_{\ell}\big\lVert \mathbf{W}^{(\ell)} \big\rVert^2

    where ∥W∥2\lVert \mathbf{W} \rVert^2 is the sum of the squares of every weight in the matrix; the biases are not penalised. The penalty's gradient is λW(ℓ)\lambda\mathbf{W}^{(\ell)}, so each weight's gradient gains λw\lambda w. With plain gradient descent the step becomes

    w←w−η(∂L∂w+λw)=(1−ηλ) w−η ∂L∂ww \leftarrow w - \eta\Big(\frac{\partial\mathcal{L}}{\partial w} + \lambda w\Big) = (1 - \eta\lambda)\,w - \eta\,\frac{\partial\mathcal{L}}{\partial w}

    so every weight shrinks by the factor 1−ηλ1 - \eta\lambda before the usual step, hence the name. λ plays the same role as ridge's λ; there the penalty was written without the ½, which only rescales λ. With Adam, as on this page, λw\lambda w is added to the gradient before Adam rescales the step, so the shrink is not exactly that factor; a variant called AdamW applies the shrink directly.

  6. The knobs

    Four numbers set how training runs. None has one right value; you watch the loss curves and choose. The table gives the values used in steps 2–3 and steps 4–5.

    Batch size: smaller batches give noisier steps but more of them per epoch (step 1). Epochs: too few and training has not finished; too many and a big network can memorise (step 4). η: how far each step goes, as in gradient descent. λ: how hard large weights are penalised (step 5).

    The last column is general guidance, not something this page showed: this page never used a learning rate that was too large or a λ that was too strong. A learning rate that is too large is shown in gradient descent.

    What to watch: the loss on points the network was not trained on. On this page that was the test set; properly it is a validation set held back for these choices, with the test set kept for one final check, as in train and test.

Quick check

What is an epoch?

That is a step. An epoch is made of steps: with 160 points in batches of 16 it is 10 of them.

Right. Each epoch shuffles the points and works through them batch by batch, using every point once. With 160 points in batches of 16 that is 10 steps.

A run is many epochs: step 3's run is 300 of them.

With the full batch an epoch happens to be one step, but an epoch means a pass through all the points, whatever the batch size. With batches of 8 it is 20 steps.

Train loss is still falling but test loss has risen for 50 epochs. What should you do?

Right. That is early stopping, as in step 4. Properly the loss to watch is on a validation set, so the test set stays untouched for a final check. Adding weight decay and training again is another option.

A falling training loss only says the network fits its training points better. In step 4 the training loss kept falling after epoch 120 while the test loss climbed to 23% above its best.

A bigger network can fit its training points even more closely, which is the problem here. In step 4 the big network got 100% of its 60 training points right while its test loss climbed.

The last weights are the ones that have been getting worse on held-back points for 50 epochs. Keep the best ones seen instead.

Lesson complete

  • Training takes many small steps, each on a batch; an epoch is one pass through all the training points.
  • Smaller batches give noisier loss curves but more steps per epoch; each batch gradient estimates the full gradient.
  • A big network can memorise its training points: watch the loss on held-back points, and stop early or add weight decay.

Common misconceptions

  • "A lower training loss means a better model." In step 4, after epoch 120 the training loss fell from 0.139 to 0.004 while the test loss rose from 0.474 to 0.584.
  • "The smoothest loss curve trains best." In step 1 the full batch gave the smoothest curve, but after the same 1600 examples it had taken 10 steps and its training loss, 0.667, was not lower than batch size 8's 0.650.
  • "Early stopping means stopping the first time the test loss goes up." In step 4 the test loss went up from one epoch to the next 45 times before reaching its lowest point at epoch 120. In practice you wait a set number of epochs and keep the best weights seen.
Next lesson: Optimisers: momentum and Adam →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 8, Section 8.1.3, "Batch and Minibatch Algorithms", on training with gradients from small batches.
  2. Goodfellow, Bengio & Courville, Deep Learning, Chapter 7, Section 7.1.1, "L2 Parameter Regularization" (weight decay), and Section 7.8, "Early Stopping".
  3. Bishop, Pattern Recognition and Machine Learning, its chapter on neural networks, which covers regularisation and early stopping.

About the data. All three sets are two interleaved spirals, made up in your browser by the mulberry32 random number generator. A set of n points has n/2 on each arm: point i of arm c (c = 0 or 1, i = 0, 1, …, n/2 − 1) has t = i/(n/2), radius 0.3 + 2.7t and angle 3πt + πc, so each arm makes one and a half turns; it sits at (radius × cos angle, radius × sin angle) plus 0.2 z on each coordinate, where z is a draw from a normal distribution with mean 0 and SD 1. Its class is its arm, c. The 160 training points use seed 171, the 400 test points seed 172, and step 4's 60 training points seed 175. Both networks start from weights drawn with seed 173, each a normal draw with mean 0 and variance 1/(number of inputs to that layer), with biases 0. Each epoch's shuffle of the training points comes from seed 174. Every run uses Adam with η = 0.01 and its usual settings (β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸); the weight-decay run differs from step 4's only in λ.

If this helped, you can .