Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Optimisers: momentum and Adam

Plain gradient descent zig-zags down a narrow valley. Momentum, RMSProp and Adam change how each step is made; here you race them, then try them on a real network.

About 11 minutes · Every path and curve is computed in your browser; the formulas are in The maths

Overview

The networks in hidden layers and training a network were trained with an optimiser called Adam. An optimiser is the rule that turns a gradient into a step. The simplest is plain gradient descent: step a fixed multiple η of the gradient downhill.

That rule struggles when the loss is a long narrow valley, steep across and gentle along, which you met in many features and the loss surface. This lesson starts with such a valley over just two weights, adds momentum, then per-weight step sizes (RMSProp), then both together (Adam), and races all four. Then it runs them on the circles network from hidden layers.

The valley is a formula, not data. The network's data are made up in your browser; see "About the data" at the end.

You'll learn

  • why plain gradient descent zig-zags in a narrow valley, and how momentum helps
  • how RMSProp and Adam give each weight its own step size, and what Adam's bias correction is for
  • how to compare optimisers fairly, and why one race does not pick a winner for every problem
  1. Back to the valley

    A loss of two weights shaped like a long narrow valley. Plain gradient descent heads for the bottom, and zig-zags.

    The loss is L=12(u2+25v2)\mathcal{L} = \tfrac{1}{2}(u^2 + 25v^2): a made-up function of two weights, not a loss on data. uu runs along the valley and vv across it. The plot's axes are the weights w1w_1 and w2w_2, and the valley is tilted 30° from them (the dotted lines). Across, the valley is 25 times as curved as along it: its curvatures are 1 and 25, so its condition number κ (from the loss surface lesson) is 25. The lowest point, ℒ = 0, is the violet square.

    Every run starts at u=−4, v=2.5u = -4,\ v = 2.5, which is (w1,w2)(w_1, w_2) = (−4.71, 0.17), where ℒ = 86.125. A run counts as there once ℒ < 10⁻³.

    At η = 0.07 the path flips from one side of the valley to the other on every step while it creeps along, and gets there in 62 steps. Move the slider. Below η = 0.04 = 1/25 it does not flip. Between 0.04 and 0.08 it flips, but each swing is smaller than the last. At exactly η = 0.08 = 2/25 the swings keep the same size for ever, and in 2000 steps ℒ never falls below 78.1. Above 0.08: Diverged: the steps grow instead of shrinking. The fastest value on the slider is 0.075, just under the limit, with 58 steps.

    The trouble is that an η big enough to move well along the valley is too big across it. The tilt is not the cause: turned to line up with w1w_1 and w2w_2, the same valley takes plain gradient descent exactly the same 62 steps at η = 0.07.

    The maths

    With u=w1cos⁡30∘+w2sin⁡30∘u = w_1\cos 30^\circ + w_2\sin 30^\circ and v=−w1sin⁡30∘+w2cos⁡30∘v = -w_1\sin 30^\circ + w_2\cos 30^\circ, the gradient-descent step w←w−η ∇L(w)\mathbf{w} \leftarrow \mathbf{w} - \eta\,\nabla\mathcal{L}(\mathbf{w}) works on uu and vv separately:

    u←(1−η) u,v←(1−25η) vu \leftarrow (1 - \eta)\,u, \qquad v \leftarrow (1 - 25\eta)\,v

    The across part shrinks only if ∣1−25η∣<1|1 - 25\eta| < 1, that is 0<η<2/250 < \eta < 2/25: the limit is 2 divided by the largest curvature. Once η>1/25\eta > 1/25 the factor is negative, so vv flips sign every step. Below the limit the along factor 1−η1 - \eta is above 0.92, so progress along the valley is slow whatever η you choose. Turning the axes changes none of this, because the step follows the gradient, and the gradient turns with the valley.

  2. Momentum

    Give the path a memory: each step uses the new gradient plus a running sum of the earlier ones, shrunk by a factor β each step.

    Here η is 0.02 and the buttons pick β. With β = 0 there is no memory: it is plain gradient descent, and it takes 223 steps. With β = 0.5 it takes 107, and with β = 0.9 only 61.

    Along the valley the gradient points the same way step after step, so its part of the running sum builds up and the path speeds up. Parts that change sign from one step to the next, like step 1's zig-zag across the valley, largely cancel in the sum. With β = 0.9 the path builds up so much speed that it passes the bottom along the valley at step 14 and has to swing back, like a ball rolling in a bowl. It swings from side to side across the valley as well, crossing the valley floor 14 times before it gets there.

    Because the steps pile up, momentum needs a smaller η than plain gradient descent: with β = 0.9 a gradient that stays the same gives a step up to 10 times as long.

    The maths

    With the gradient g=∇L(w)\mathbf{g} = \nabla\mathcal{L}(\mathbf{w}) and a running sum m\mathbf{m} that starts at 0\mathbf{0}, each step is

    m←β m+g,w←w−η m\mathbf{m} \leftarrow \beta\,\mathbf{m} + \mathbf{g}, \qquad \mathbf{w} \leftarrow \mathbf{w} - \eta\,\mathbf{m}

    Written out, mk=gk+β gk−1+β2gk−2+⋯\mathbf{m}_k = \mathbf{g}_k + \beta\,\mathbf{g}_{k-1} + \beta^2\mathbf{g}_{k-2} + \cdots. If every gradient were the same g\mathbf{g}, this would grow towards g/(1−β)\mathbf{g}/(1 - \beta), 10 times g\mathbf{g} for β = 0.9. If the gradient flipped sign every step with the same size, the sum would settle at a size of ∥g∥/(1+β)\lVert\mathbf{g}\rVert/(1 + \beta), about half of one gradient. Consistent directions add up; flipping ones cancel. Some books scale the sum by 1−β1 - \beta; that only rescales η.

  3. Per-weight step sizes

    RMSProp keeps a running average of each weight's squared gradient and divides that weight's step by its square root, so a weight with large gradients has its step scaled down more than one with small gradients, and their steps end up of similar size.

    With η = 0.1 (its best value on the grid in step 5) RMSProp gets there in 65 steps.

    Look at its first step, a big jump. The running average starts at 0, so after one step it is only 0.001 times the squared gradient, and dividing by its square root makes each weight move 3.16, as the read-out says. After that the average fills up and the steps settle.

    RMSProp scales each weight on its own, that is, along the plot's axes w1w_1 and w2w_2. This valley is tilted from those axes, so its steep direction mixes both weights, and per-weight scaling can only partly undo its shape. Turned to line up with w1w_1 and w2w_2, the same valley takes the same RMSProp run just 4 steps.

    The maths

    With s\mathbf{s} starting at 0\mathbf{0}, each step is

    s←β2 s+(1−β2) g2,w←w−η gs+ϵ\mathbf{s} \leftarrow \beta_2\,\mathbf{s} + (1 - \beta_2)\,\mathbf{g}^2, \qquad \mathbf{w} \leftarrow \mathbf{w} - \eta\,\frac{\mathbf{g}}{\sqrt{\mathbf{s}} + \epsilon}

    The square, the square root and the division are taken weight by weight. This page uses β2=0.999\beta_2 = 0.999, the value Adam uses below, and ϵ=10−8\epsilon = 10^{-8}, which only guards against dividing by 0. On the first step s=0.001 ∣g∣\sqrt{s} = \sqrt{0.001}\,|g| for each weight, so each weight moves η/0.001≈31.6 η\eta/\sqrt{0.001} \approx 31.6\,\eta, whatever the size of its gradient.

  4. Adam

    Adam does both: a running average of the gradient, as in momentum, and of its square, as in RMSProp, each corrected for starting at zero.

    With η = 1, its best value on the grid, Adam gets there in 65 steps. The averaged gradient plays the part of momentum's running sum; the averaged square sets each weight's step size.

    Both averages start at 0, so without a correction they would be far too small at first. Adam divides each by a factor that makes up for this, the bias correction. Its effect shows on the first step: each weight moves η, 1.00 here, as the read-out says. The correction fades as the steps go on.

    The maths

    With m\mathbf{m} and s\mathbf{s} starting at 0\mathbf{0}, step t=1,2,…t = 1, 2, \ldots is

    m←β1m+(1−β1) g,s←β2s+(1−β2) g2\mathbf{m} \leftarrow \beta_1\mathbf{m} + (1 - \beta_1)\,\mathbf{g}, \qquad \mathbf{s} \leftarrow \beta_2\mathbf{s} + (1 - \beta_2)\,\mathbf{g}^2
    m^=m1−β1 t,s^=s1−β2 t,w←w−η m^s^+ϵ\hat{\mathbf{m}} = \frac{\mathbf{m}}{1 - \beta_1^{\,t}}, \qquad \hat{\mathbf{s}} = \frac{\mathbf{s}}{1 - \beta_2^{\,t}}, \qquad \mathbf{w} \leftarrow \mathbf{w} - \eta\,\frac{\hat{\mathbf{m}}}{\sqrt{\hat{\mathbf{s}}} + \epsilon}

    with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999 and ϵ=10−8\epsilon = 10^{-8}. Why the correction: if every gradient were the same g\mathbf{g}, then after tt steps m=(1−β1 t) g\mathbf{m} = (1 - \beta_1^{\,t})\,\mathbf{g}, short of g\mathbf{g} because it started at 0; dividing by 1−β1 t1 - \beta_1^{\,t} gives back g\mathbf{g}, and the same goes for s\mathbf{s}. At t=1t = 1, m^=g\hat{\mathbf{m}} = \mathbf{g} and s^=g2\hat{\mathbf{s}} = \mathbf{g}^2, so each weight moves η (to within ε). Without the correction the first step would be 0.1/0.001≈3.160.1/\sqrt{0.001} \approx 3.16 times η. As tt grows, β1 t\beta_1^{\,t} and β2 t\beta_2^{\,t} go to 0 and the correction fades. Books use other letters for m\mathbf{m} and s\mathbf{s}.

  5. The race

    All four from the same start, each with its own η picked from the same grid. Press Train.

    Play steps through the lesson, and Train starts the race. All four start on the same point, so before Train only one marker shows. Each ball takes one step every two frames and stops once its ℒ is below 10⁻³.

    Picking η fairly: each optimiser was run with every η in the grid 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5 and 1, for up to 2000 steps, and kept the value that got there in the fewest steps. That gave η = 0.05 for plain gradient descent, 0.02 for momentum (β = 0.9), 0.1 for RMSProp and 1 for Adam.

    On this valley, from this start, momentum got there first, in 61 steps; RMSProp and Adam were level at 65 and 65; plain gradient descent took 88.

    Read that narrowly. The grid is coarse. Plain gradient descent's grid value is not its best: step 1's η = 0.075 took 58 steps, fewer than momentum's 61. Adam's best was the largest η on the grid; at η = 2, just off it, Adam takes 43. A finer or wider grid would change the order.

    The maths

    For each optimiser and each η in the grid GG, run from the start until L<10−3\mathcal{L} < 10^{-3} or 2000 steps, and keep

    η⋆=arg⁡min⁡η∈G (steps to reach L<10−3)\eta^\star = \arg\min_{\eta \in G}\ (\text{steps to reach } \mathcal{L} < 10^{-3})

    with ties going to the smaller η. Counting steps is a fair measure of work here: every optimiser uses one gradient per step, and the extra arithmetic for the running averages is small next to computing a gradient in a real network.

  6. On a real network

    The same four optimisers on the circles network from hidden layers: two inputs, one hidden layer of 4 tanh units, one output.

    Each run has the same 200 training points, the same starting weights and the same batches of 20, for 100 epochs. Each η was picked from the same grid, this time by the lowest training loss averaged over the 100 epochs, which rewards getting down fast and staying down: 1 for plain gradient descent, 0.5 for momentum, 0.05 for RMSProp and 0.1 for Adam. The four runs are computed when you first reach this step, which takes a moment.

    The curves are the training loss after each epoch: binary cross-entropy, the log-loss of logistic regression, averaged over the training points. After 100 epochs it is 0.020 for momentum, 0.044 for Adam, 0.059 for RMSProp and 0.193 for plain gradient descent. On the 400 test points they get 98.0%, 97.5%, 95.5% and 90.5% right.

    RMSProp and Adam, level on the valley, differ here: Adam ends lower. Plain gradient descent's best η was the largest on the grid. Every curve goes up between some epochs: each step uses a batch of 20 points, so the steps are noisy, as in training a network, and a large step can also overshoot: with all 200 points in every batch, the same RMSProp run still goes up between 40 pairs of epochs.

    Rankings like these change from problem to problem, and with the grid, the batch size and the starting weights. To choose an optimiser for a new problem, try a few, tune η for each, and judge them on held-back data.

    The maths

    The network is a(1)=tanh⁡(W(1)x+b(1))\mathbf{a}^{(1)} = \tanh\big(\mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}\big), p^=σ(W(2)a(1)+b(2))\hat{p} = \sigma\big(\mathbf{W}^{(2)}\mathbf{a}^{(1)} + b^{(2)}\big), with 17 weights and biases. σ here is the sigmoid, not a standard deviation. ℒ for a batch is the mean log-loss over its 20 points; its gradient comes from backpropagation, and the optimiser turns it into a step, exactly as on the valley. An epoch is 10 steps, so each run is 1000 steps.

Quick check

Why does plain gradient descent zig-zag here?

Right. Across the valley each step multiplies v by 1 − 25η, which is negative once η is above 1/25, so v flips sign every step. Along it, the factor 1 − η is close to 1, so progress is slow.

Plain gradient descent does not depend on the tilt. Lined up with the axes, the same valley took it the same 62 steps at η = 0.07, zig-zagging just the same. The tilt matters to RMSProp, which scales each weight separately.

The zig-zag comes from η being too large for the steep direction: below η = 0.04 the path does not flip at all.

On the valley each step uses the exact gradient of ℒ; there are no data and no batches. The zig-zag comes from the valley's shape.

What does Adam's bias correction fix?

Right. After t steps the averages are short by the factors 1 − β₁ᵗ and 1 − β₂ᵗ because they started at 0; dividing by those factors puts that right. On the valley Adam's first step moved each weight by η.

The averaging itself smooths noisy gradients. The correction only rescales the averages, and its effect fades as the steps go on.

The correction matters most at the start, not near the end: its factors go to 1 as the steps go on.

The correction does nothing about direction. Adam scales each weight separately, along the axes, with or without the correction.

Lesson complete

  • In a narrow valley, an η that moves well along it overshoots across it, so plain gradient descent zig-zags; it diverges once η passes 2 divided by the largest curvature.
  • Momentum adds up steps that agree and lets flipping ones cancel; RMSProp gives each weight its own step size; Adam does both, with a correction for starting at zero.
  • Compare optimisers with each one's η tuned, and judge on the problem at hand: on this page the order changed with the grid, and between the valley and the network.

Common misconceptions

  • "Adam always wins." Not on this page: on the valley momentum got there in 61 steps against Adam's 65, and on the network momentum's training loss ended at 0.020 against Adam's 0.044, with the grids and settings stated. On other problems the order can be different again.
  • "Giving each weight its own step size fixes any narrow valley." On this page it helped most when the valley lined up with the axes: RMSProp took 65 steps on the tilted valley and 4 on the same valley lined up with the axes.
  • "Plain gradient descent zig-zags because the valley is tilted." It took 62 steps whether the valley was tilted or not. The zig-zag comes from the difference in curvature.
Next lesson: Going deeper: vanishing gradients →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 8, Section 8.3.2, "Momentum", and Section 8.5, "Algorithms with Adaptive Learning Rates", which covers RMSProp (8.5.2) and Adam (8.5.3).
  2. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, its chapter on continuous optimisation, on gradient descent and momentum.

About the data. The valley in steps 1 to 5 is the formula ℒ = ½(u² + 25v²) with u = w₁ cos 30° + w₂ sin 30° and v = −w₁ sin 30° + w₂ cos 30°, started from u = −4, v = 2.5; nothing in it is random. The network in step 6 uses the data of the hidden-layers lesson, made up in your browser by the mulberry32 random number generator. Each point gets an angle θ drawn uniformly from 0 to 2π and a radius r = r₀ + 0.25 z, where z is a draw from a normal distribution with mean 0 and SD 1; the point is (r cos θ, r sin θ). Class 0 has r₀ = 1 and class 1 has r₀ = 2.2, half the points each. The 200 training points use seed 151 and the 400 test points seed 152. The network's starting weights are drawn with seed 155 (normal, variance 1 divided by the number of inputs to the layer; biases start at 0), and the order of the training points in each epoch is shuffled with seed 154; all four runs share both. The optimisers use β = 0.9 (momentum), β₁ = 0.9, β₂ = 0.999 (β₂ also for RMSProp) and ε = 10⁻⁸ throughout.

If this helped, you can .