Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Convolutional networks

Stack a convolution, a ReLU and a pooling step, put one output unit on top, and let training choose the filters. With weights shared across the image, the whole network needs far fewer of them than a dense one.

About 11 minutes · Every map, run and count is computed in your browser; the formulas are in The maths

Overview

In convolutions the filters were typed in by hand. Here a tiny convolutional network (CNN) starts with two random 3 × 3 filters and adjusts them, along with everything else, to lower its loss on the training images. That is all "learns" means here.

The task: tell a plus-shaped cross from a hollow square, each drawn somewhere on an 8 × 8 image with speckled noise. You will see each layer's maps for one image, what pooling does when the image moves by a pixel, the filters changing while it trains, and how many weights it needs next to a dense network trained the same way.

You'll learn

  • how convolution, ReLU, max-pooling and one output unit fit together, and the size of each layer
  • that training adjusts the filters themselves, and what these two ended up as
  • why sharing weights gives 39 weights and biases where a dense network on the same pixels has 1,189
  1. Two shapes, shifted and noisy

    Each image is 8 × 8 pixels. Class 0 is a cross of five bright pixels; class 1 is the eight-pixel outline of a 3 × 3 square, with a dark centre.

    These are the first 12 of the 200 training images. Every shape is placed at a random position and every pixel gets random noise, so the images differ in where the shape sits and in their speckle. Crosses carry a circle marker and squares a triangle.

    A separate set of 400 test images, made the same way with a different seed, is used only to check the network, as in train and test.

    The maths

    Each image is a matrix XX with 8 rows and 8 columns, values in [0,1][0, 1]. Its label is y=0y = 0 for a cross and y=1y = 1 for a square. The exact recipe is in "About the data" at the end.

  2. The layers

    The image goes through four stages: convolution with two filters, ReLU, 2 × 2 max-pooling, and one output unit that sees the 18 pooled values.

    Each 3 × 3 filter slides over the image as in the last lesson, giving a 6 × 6 map. ReLU sets every negative value to 0. Pooling keeps the largest value in each 2 × 2 block, so each map shrinks to 3 × 3. The two pooled maps make 18 numbers, and the output unit turns them into a probability that the image is a square.

    These are the network's starting filters, drawn at random, before any training. For example 1, a cross, the output is p^\hat{p} = 0.76: a guess, and a wrong one. Pick another example to see its maps. Cyan is positive; violet is negative and carries a minus sign.

    The maths

    Convolution here is cross-correlation (no flip), with stride s=1s = 1 and padding p=0p = 0. Filter kk has weights Wk\mathbf{W}_k and a bias bkb_k:

    Ak[r,c]=ReLU(bk+∑i=02∑j=02X[r+i, c+j] Wk[i,j]),r,c=0,…,5A_k[r, c] = \mathrm{ReLU}\Big(b_k + \sum_{i=0}^{2}\sum_{j=0}^{2} X[r + i,\, c + j]\,\mathbf{W}_k[i, j]\Big),\quad r, c = 0, \dots, 5

    The size follows ⌊(n+2p−f)/s⌋+1=⌊(8+0−3)/1⌋+1=6\lfloor (n + 2p - f)/s \rfloor + 1 = \lfloor (8 + 0 - 3)/1 \rfloor + 1 = 6. Pooling takes Mk[r,c]=max⁡(Ak[2r,2c], Ak[2r,2c+1], Ak[2r+1,2c], Ak[2r+1,2c+1])M_k[r, c] = \max(A_k[2r, 2c],\, A_k[2r, 2c{+}1],\, A_k[2r{+}1, 2c],\, A_k[2r{+}1, 2c{+}1]) for r,c=0,1,2r, c = 0, 1, 2. The 18 values of M1M_1 and M2M_2, row by row, form a vector m\mathbf{m}, and

    z=v⋅m+bout,p^=σ(z)z = \mathbf{v}\cdot\mathbf{m} + b_{\text{out}}, \qquad \hat{p} = \sigma(z)

    where p^\hat{p} is the network's probability that the image is a square, and σ here is the sigmoid, not a standard deviation. The shapes, layer by layer: 8×8→6×6×2→3×3×2→18→18 \times 8 \to 6 \times 6 \times 2 \to 3 \times 3 \times 2 \to 18 \to 1. The loss for one image is the binary cross-entropy, which is the log-loss of logistic regression: L=−yln⁡p^−(1−y)ln⁡(1−p^)\mathcal{L} = -y \ln \hat{p} - (1 - y)\ln(1 - \hat{p}).

  3. Pooling

    2 × 2 max-pooling keeps the strongest response in each block and halves each side of the map. If the image moves a little, the strongest response can stay in the same block.

    The top row is example 1; the bottom row is the same image moved one pixel to the right, with a column of 0s coming in on the left. Lines mark the 2 × 2 blocks, and a ring marks the largest value in each block (when it is above 0).

    For example 1, with the starting filters, the maps after ReLU change by 82% of their total, but the pooled maps change by only 46%, and 5 of the 18 pooled values do not change at all: in those blocks the largest value just moves within its block, so the pooled value is the same. The pooled change is the smaller one for each of the 12 examples from step 1 too; pick another to see its numbers. Pooling makes small shifts matter less; it does not make them vanish.

    The maths

    The change is measured as ∑∣a−a′∣ / ∑∣a∣\sum |a - a'| \,/\, \sum |a| over all the values of the two maps, where aa is a value for the image and a′a' the same value for the shifted image.

    In backpropagation, each pooled value Mk[r,c]M_k[r, c] equals exactly one of the four values in its block, the largest. So ∂Mk[r,c]/∂Ak[i,j]\partial M_k[r, c] / \partial A_k[i, j] is 1 for that cell and 0 for the other three: the gradient through max-pooling goes only to the max. If two cells tie, this code passes it to the first in row order. The ReLU then passes the gradient on only where the value before ReLU was above 0, and the filter's gradient adds up the 3 × 3 patches under those cells:

    ∂L∂Wk[i,j]=∑r,c∂L∂Mk[r,c]  1[Ak[r∗,c∗]>0]  X[r∗+i, c∗+j]\frac{\partial \mathcal{L}}{\partial \mathbf{W}_k[i, j]} = \sum_{r, c} \frac{\partial \mathcal{L}}{\partial M_k[r, c]}\; \mathbb{1}\big[A_k[r^*, c^*] > 0\big]\; X[r^* + i,\, c^* + j]

    where (r∗,c∗)(r^*, c^*) is the position of the max in block (r,c)(r, c). Pooling itself has no weights.

  4. Train it

    Press Train. The two filters change in front of you, and the read-out shows the training loss and the test accuracy after each epoch.

    Play steps through the lesson, and Train trains the network. Training runs Adam (from optimisers) with η = 0.02 on batches of 20 images for 30 epochs; one epoch is one pass through the 200 training images.

    These numbers are for the whole run, to the end of epoch 30. At the start the training loss ℒ is 0.743 and test accuracy is 50%, a coin toss. After epoch 6 the test accuracy is 92.5%; after epoch 7 it is 99.75%, and it stays at or above 95% from then on. At the end the loss is 0.029 and every one of the 400 test images is right: 100%.

    The weights shown in each filter are its current values. Pick an example to watch its two maps change as the filters do. Reset goes back to the same starting filters, and the next run is the same as the last.

    The maths

    Each step takes the mean loss over a batch of 20 images and computes its gradient for all 39 weights and biases by backpropagation, through the dense output, the pooling (to the max only), the ReLU and the convolution. Adam then moves each one, with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ε=10−8\varepsilon = 10^{-8}. The training loss in the read-out is the mean over all 200 training images at the end of each epoch.

    A filter weight's gradient is a sum over every position the filter was used at, because the same weight is used at all of them. That is weight sharing, seen from the training side.

  5. What the filters became

    The filters at the end of the run, and their maps after ReLU on a cross (example 1) and a square (example 2).

    Filter 1 is positive along its top and bottom rows and negative along its middle row, most of all at the centre, −1.43. Filter 2 has the same pattern turned on its side: positive down its left and right columns, negative down the middle, with −1.36 at the centre.

    Laid over the square's centre, filter 1's positive top and bottom rows sit on the square's top and bottom edges; its side weights, −0.47 and −0.48, sit on the square's left and right edges and take a little back; and its centre weight sits on the dark middle, which is near 0. With the filter's bias of −1.43 added, the sum is still large: both maps are largest at that position, 2.46 and 2.52. On this cross every value of both maps is below 0 before ReLU, so after ReLU both maps are all 0: their largest values are 0.00 and 0.00. On these two examples, each filter acts like a square detector.

    With all 18 pooled values at 0, the cross's output is the output bias alone: p^\hat{p} = 0.044. All 18 output weights ended positive, so any response from the filters pushes the output towards "square"; the network labels this image a cross because neither filter responds. For the square p^\hat{p} is 1.000 to three decimals. Nobody told the network to look for outlines; these weights are where training ended up from random filters.

    The maths

    For the cross, every entry of m\mathbf{m} is 0, so z=v⋅0+bout=boutz = \mathbf{v}\cdot\mathbf{0} + b_{\text{out}} = b_{\text{out}} and p^=σ(bout)\hat{p} = \sigma(b_{\text{out}}). Another starting seed could end at different filters; the description here is of this run.

  6. Fewer weights

    The CNN has 39 weights and biases. A dense network from the 64 pixels to 18 hidden units and one output has 1,189, about 30 times as many.

    The dense network was trained the same way: Adam, η = 0.02, batches of 20, 30 epochs, the same shuffle seed. It fits the training images completely (100% right, with a final training loss of 0.003, lower than the CNN's 0.029), but on the test images it gets 92.75%: 29 of the 400 wrong. The CNN gets 100%.

    That gap between training and test is overfitting. A likely reason, not tested here: the dense network has its own weight for every pixel, so what it picks up about a cross in one place does not automatically carry over to a cross one pixel over. The CNN uses the same filter weights at every position.

    The maths

    CNN: two filters of 9 weights (18), two filter biases, 18 output weights and one output bias: 2×9+2+18+1=392 \times 9 + 2 + 18 + 1 = 39.

    Dense: W(1)\mathbf{W}^{(1)} is 18 × 64 (1,152 weights), b(1)\mathbf{b}^{(1)} has 18, W(2)\mathbf{W}^{(2)} 18 and b(2)b^{(2)} one: 64×18+18+18+1=118964 \times 18 + 18 + 18 + 1 = 1189. Its hidden units use ReLU, a(1)=ReLU(W(1)x+b(1))\mathbf{a}^{(1)} = \mathrm{ReLU}(\mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}), where x\mathbf{x} is the 64 pixels in one long vector, row by row. Both counts are computed on the page from the two networks' sizes.

    This is one data set and one run each; a bigger or differently shaped dense network, or another seed, would give other numbers.

Quick check

What does max-pooling pass back during backprop?

Right. The pooled value is that one cell, so only that cell affects the loss through it; a small change to any of the other three leaves the max unchanged.

That is what average pooling does, because there every cell counts a quarter. Max-pooling's output is just the largest cell.

Pooling has no weights of its own, but the gradient still has to pass through it to reach the filters, or they could never be trained.

A small change to any of the other three leaves the max, and so the output, unchanged, so their gradient is 0.

Why does the CNN need fewer weights than the dense net?

Right. 18 shared filter weights and 2 biases cover the whole image, and the output needs only 18 weights and a bias: 39 in all.

Both were trained for 30 epochs. The number of weights is fixed by the shape of the network before any training.

Their size does not change how many there are. The count comes from the network's shape.

It has more stages (convolution, ReLU, pooling, output) than the dense net's hidden layer and output, and still fewer weights.

Lesson complete

  • A CNN here is convolution, ReLU, 2 × 2 max-pooling and one output unit: 8 × 8 → 6 × 6 × 2 → 3 × 3 × 2 → 18 → 1.
  • Training adjusts the filters too: from random starts these two ended with positive outer rows or columns and a negative centre, and the network got all 400 test images right.
  • Sharing weights gave 39 weights and biases against the dense network's 1,189, and here the CNN did better on the test images.

Common misconceptions

  • "Pooling layers have weights to learn." They have none; all 39 numbers here belong to the filters, their biases and the output unit.
  • "Fewer weights means a weaker model." On this task the 39-weight CNN got 100% of the test images right and the 1,189-weight dense network 92.75%.
  • "Pooling makes a CNN ignore where things are." For example 1 a one-pixel shift still changed the pooled maps by 46%; pooling reduces the effect of small shifts, it does not remove it.
  • "Someone chose the filters." They started as random numbers and were adjusted by training to lower the loss on the training images.
Next lesson: Recurrent networks →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 9, "Convolutional Networks", on convolution, pooling and weight sharing.
  2. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), on convolutional neural networks.

About the data. All images are 8 × 8, made up in your browser by the mulberry32 random number generator, and classes alternate: image 0 is a cross (class 0), image 1 a square (class 1), and so on. A cross is the centre pixel and its four neighbours (up, down, left, right); a square is the eight pixels around the centre, with the centre left at 0. For each image the generator gives, in order, the centre row and then the centre column, each 1 + ⌊6u⌋ for a uniform draw u, so the whole 3 × 3 shape fits inside the image; then one normal draw z (mean 0, SD 1) per pixel, row by row. Each pixel is 1 on the shape or 0 off it, plus 0.15 z, clipped to [0, 1]. The 200 training images use seed 201 and the 400 test images seed 202. The CNN's starting weights use seed 203: the filter weights first, each a normal draw with mean 0 and variance 2/9, then the 18 output weights, variance 1/18; biases start at 0. Each epoch's shuffle of the training images comes from seed 204. The dense network (64 → 18 ReLU units → 1) starts from seed 205, each weight a normal draw with mean 0 and variance 2/(number of inputs to that layer), biases 0, and is trained with the same shuffle seed. Both use Adam with η = 0.02, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, batches of 20 and 30 epochs.

If this helped, you can .