Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Convolutions

A filter is a small grid of weights slid across an image. At each position it multiplies the pixels under it by its weights and adds them up, giving one number, large where the patch looks like the filter. The same nine weights are used everywhere.

About 10 minutes · Every map and sum is computed in your browser; the formulas are in The maths

Overview

The networks in hidden layers and going deeper connect every input to every unit. For an image that wastes weights: a vertical edge looks the same near the top corner as in the middle, so why learn to spot it separately in each place?

A convolution answers that with one small filter, here 3 × 3, slid over the whole image. This lesson does it step by step on a 12 × 12 image made up in your browser (see "About the data" at the end). Nothing is trained here; the next lesson lets a network learn its own filters.

You'll learn

  • how a filter turns each 3 × 3 patch of an image into one number of a feature map
  • how stride and padding set the size of the map, ⌊(n + 2p − f)/s⌋ + 1
  • why using the same weights at every position finds a pattern wherever it is, with far fewer weights
  1. An image is a grid of numbers

    To the computer this picture is 144 numbers in 12 rows and 12 columns: 0 is dark, 1 is bright and 0.5 is grey.

    There is a bright 4 × 4 square near the top left and a grey diagonal line lower down; every other pixel is 0. Move the cursor with the row and column sliders, or hover or tap on the image, and the panel on the right shows the numbers around it, as in data as numbers.

    Rows and columns are counted from 0, from the top left, here and in the rest of the lesson.

    The maths

    The image is a matrix XX with entries X[r,c]X[r, c] for r,c=0,1,…,11r, c = 0, 1, \ldots, 11. A colour image would have three such grids (red, green and blue); this one has a single grey channel.

  2. A filter, at one position

    The filter W is a 3 × 3 grid of weights. Lay it over a 3 × 3 patch, multiply each pixel by the weight on top of it, and add the nine products.

    This filter has −1 down its left column, 0 in the middle and +1 on the right, so the sum is the right column of the patch minus the left column. It is large when the patch is dark on the left and bright on the right: a vertical edge.

    The window starts at map row 2, column 0, which covers image rows 2–4 and columns 0–2. Its right column is inside the square (three 1s) and its left column is background (three 0s), so the sum is 3. Move the window with the sliders, or tap the image, and the products and the sum follow.

    The maths

    For the window at row rr, column cc (stride 1, no padding):

    (X⋆W)[r,c]=∑i=02∑j=02X[r+i, c+j] W[i,j](X \star \mathbf{W})[r, c] = \sum_{i=0}^{2}\sum_{j=0}^{2} X[r + i,\, c + j]\,\mathbf{W}[i, j]

    Convolution here is cross-correlation (no flip): the filter is laid on the patch as written. The convolution of signal processing first flips the filter top to bottom and left to right, ∑i,jX[r−i, c−j] W[i,j]\sum_{i,j} X[r - i,\, c - j]\,\mathbf{W}[i, j]. For a filter that a network learns, the flip makes no difference: it would simply learn the flipped weights.

  3. Slide it everywhere

    Do the same at every position, left to right and top to bottom. Each sum becomes one cell of the feature map.

    A 3 × 3 window fits in 10 positions across a row of 12 pixels, so the map is 10 × 10. Positive values are cyan; negative ones are violet and carry a minus sign. "Slide again" replays the sweep.

    The largest value, 3, is at map rows 2–3, columns 0–1: windows whose three rows sit in the square and that straddle its left edge. The smallest, −3, is at rows 2–3, columns 4–5, on the right edge, where bright turns to dark. Inside the square and on the plain background, left and right cancel and the map is 0. Windows that hold no pixel of the square see only the diagonal line, and give at most 0.5 either way.

    The maths

    Here W=(−101−101−101)\mathbf{W} = \begin{pmatrix} -1 & 0 & 1 \\ -1 & 0 & 1 \\ -1 & 0 & 1 \end{pmatrix}, so each map value is ∑i=02(X[r+i, c+2]−X[r+i, c])\sum_{i=0}^{2} \big(X[r+i,\, c+2] - X[r+i,\, c]\big), a sum over three rows of the difference between pixels two columns apart. With pixels between 0 and 1 it lies between −3 and 3, and it reaches 3 only when the right column is all 1 and the left column all 0.

  4. Other filters

    Pick a different filter, or type your own nine weights, and the map is recomputed.

    The horizontal-edge filter is the vertical one turned on its side. Its largest value, 3, is at rows 0–1, columns 2–3, on the square's top edge, and its smallest, −3, at rows 4–5, columns 2–3, on the bottom edge. Blur has every weight 1/9, so each value is the mean of the nine pixels: 1 where the window is inside the square, softer at its edges, and at most 0.17 along the thin line.

    "noisy" swaps in a speckled copy of the image. The vertical-edge filter still gives its largest value, 2.95, on the left edge, at row 3, column 0; where the clean image's map is 0, the noisy one stays within 0.51 of 0.

    "your own" opens nine boxes under the picture. Each takes a number from −2 to 2 in steps of 0.5; anything else is moved to the nearest allowed value when you leave the box. "Try the line filter" fills in 1 from top right to bottom left and −0.5 elsewhere. That filter looks like the line, and its map is largest along it: 1.5 there, against at most 1 on windows that touch the square.

    The maths

    Blur is W[i,j]=19\mathbf{W}[i, j] = \tfrac19 for every i,ji, j, so (X⋆W)[r,c]=19∑i,jX[r+i, c+j](X \star \mathbf{W})[r, c] = \tfrac19 \sum_{i,j} X[r + i,\, c + j], the mean of the patch. Each map value is a weighted sum, so the map is linear in the weights: doubling every weight doubles every value, and the map of the sum of two filters is the sum of their maps.

  5. Stride and padding

    The stride s is how far the window moves each time; padding p adds a ring of zeros around the image. Together they set the size of the map.

    With s = 1 and p = 0 the map is 10 × 10. Stride 2 skips every other position: 5 × 5. Padding 1 lets the window sit on the border, centred on each pixel in turn, so with stride 1 the map is 12 × 12, the same size as the image; with stride 2 it is 6 × 6. The dashed ring marks the zeros. Move the window to row 0, column 0 with the sliders and part of it sits on the ring: those pixels count as 0, and so do their products.

    The read-out works the size out from the formula, and the map drawn is counted from the actual sliding; for every setting here the two agree.

    The maths

    With stride ss, padding pp and X[r,c]=0X[r, c] = 0 outside the image,

    (X⋆W)[r,c]=∑i=0f−1∑j=0f−1X[rs+i−p,  cs+j−p] W[i,j](X \star \mathbf{W})[r, c] = \sum_{i=0}^{f-1}\sum_{j=0}^{f-1} X[r s + i - p,\; c s + j - p]\,\mathbf{W}[i, j]

    For an n×nn \times n image and an f×ff \times f filter, the window's top row can start at −p,−p+s,−p+2s,…-p, -p + s, -p + 2s, \ldots as long as its bottom row is no lower than n−1+pn - 1 + p, which gives

    ⌊n+2p−fs⌋+1\left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1

    rows, and as many columns. Here n=12n = 12 and f=3f = 3.

  6. The same weights everywhere

    The whole map came from just 9 weights, used again at every position. Shift the image and the map shifts with it.

    This is weight sharing. A layer that joined every one of the 144 pixels to every one of the 100 map values with its own weight would need 14,400 weights. The filter needs 9, because an edge is the same pattern wherever it is.

    Shift the image right by 2 and the largest value moves to rows 2–3, columns 2–3, two columns to the right, with the same value. This holds away from the edges: columns that come in from the left are new zeros, and anything pushed off the right edge is lost. Here no pixel is pushed off the image, so every value that stays on the map moves exactly two columns right; the map's last two columns slide out of view and two columns of zeros come in on the left.

    A convolutional network stacks layers like this one and learns the weights of its filters instead of having them typed in. That is the next lesson.

    The maths

    If X′[r,c]=X[r,c−d]X'[r, c] = X[r, c - d] (shifted right by dd), then with stride 1

    (X′⋆W)[r,c]=∑i,jX[r+i, c−d+j] W[i,j]=(X⋆W)[r,c−d](X' \star \mathbf{W})[r, c] = \sum_{i,j} X[r + i,\, c - d + j]\,\mathbf{W}[i, j] = (X \star \mathbf{W})[r, c - d]

    whenever every pixel the window covers came from inside the original image. This is called equivariance to translation. With padding, the windows in the last column also change, because part of what they covered has been pushed off. With stride 2, a shift of one pixel does not move the map by a whole cell, so there it holds only for even shifts. The dense layer's count is 144×100=14 400144 \times 100 = 14\,400 weights, before biases.

Quick check

A 12 × 12 image, a 3 × 3 filter, stride 1, no padding: what size is the feature map?

Right. ⌊(12 + 0 − 3)/1⌋ + 1 = 10: the window's left column can sit on columns 0 to 9.

That needs padding 1, so the window can be centred on the border pixels. Without padding the window must fit inside the image, which leaves 10 positions.

12 − 3 = 9 counts the moves, not the positions: starting at column 0 and moving 9 times gives 10 positions.

That would be 12 ÷ 3, as if the windows did not overlap. With stride 1 they move one pixel at a time and overlap, so there are 10 positions per row.

Why use the same filter weights at every position?

Right. Here 9 shared weights made the whole map, where a dense layer would need 14,400 weights to connect the same pixels to the same map values; and shifting the image shifted the map.

The map's size comes from n, f, s and p, not from sharing the weights.

Each map value sees only its 3 × 3 window. Sharing means the same weights are used for every window, not that a window sees more.

The weights are the same, but the patches differ, so the values differ: 3 on the left edge, 0 on flat patches.

Lesson complete

  • A filter is a small grid of weights; at each position the map value is the sum of the pixels times the weights, large where the patch looks like the filter.
  • The map has ⌊(n + 2p − f)/s⌋ + 1 rows and columns: 10 × 10 here with stride 1 and no padding.
  • The same 9 weights are used at every position, so a pattern is found wherever it is, and shifting the image shifts the map, away from the edges.

Common misconceptions

  • "The feature map is a smaller copy of the image." It is the filter's response: inside the bright square the vertical-edge map is 0, and it reaches 3 and −3 only on the square's sides.
  • "Convolution in a network flips the filter, as in a signal-processing textbook." Here it is cross-correlation: the filter is laid on the patch as written.
  • "Padding adds information about the border." It adds zeros, so the window can sit on the border pixels; with stride 1 and padding 1 the map stays 12 × 12.
Next lesson: Convolutional networks →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 9, "Convolutional Networks", on the convolution operation and weight sharing.
  2. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), on convolutional neural networks.

About the data. The image is 12 × 12, with rows and columns counted from 0. Every pixel is 0 except a 4 × 4 square of 1s in rows 2–5 and columns 2–5, and a diagonal line of 0.5s on the five pixels with row + column = 15, from row 6, column 9 to row 10, column 5. The noisy copy adds Gaussian noise with mean 0 and standard deviation 0.1 to every pixel, one draw per pixel in row order, from the mulberry32 random number generator with seed 191, and then clips each pixel to [0, 1], so the background can only get brighter and the square only darker. The shifted image in the last step moves every pixel right and fills the new columns on the left with 0.

If this helped, you can .