Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

Hidden layers: bending the boundary

Press Train and watch a small network bend its boundary, step by step, around data that no straight line can split.

About 11 minutes · No maths needed; the maths is there if you want it

Overview

In the previous lesson one neuron could only draw a straight line, and two hand-wired neurons drew two lines that a third combined. This lesson lets a network find its own lines.

The data are two rings, one inside the other. You press Train and the network adjusts its weights, a small step at a time, to lower its loss on the training points. Its boundary bends, step by step, until it wraps around the inner ring. Every number on the stage is the network's real state at that moment: nothing is eased, faked or replayed.

The data are made up. See "About the data" at the end.

You'll learn

  • how a hidden layer lets a network bend its boundary: each hidden unit draws a line and the output unit combines them
  • what training does, watched live: the loss falls as the weights move
  • why the activation must be non-linear, and what the hidden layer does to the data
  1. One straight line can't

    Two rings of points, class 0 inside and class 1 outside. One neuron draws one straight line, and here that is no use.

    There are 200 training points: cyan circles (class 0) on the inner ring and violet triangles (class 1) on the outer one. Another 400 test points from the same rings are kept aside and not drawn; they measure how well a boundary works on points it never saw.

    The line is one sigmoid neuron fitted as logistic regression (the previous lesson showed they are the same). It gets 57.0% of the training points right and 49.8% of the test points. Half the test points are each class, so that is no better than a coin toss.

    The fit makes the log-loss small, not the count of mistakes, but no straight line does well here. Checking every line, as the previous lesson did, the best gets 71.5% of the training points right, by cutting a slice off the outer ring. The best lines leave the whole inner ring on one side, along with part of the outer one.

    The maths

    One sigmoid neuron gives

    p^=σ(w⋅x+b)\hat{p} = \sigma(\mathbf{w}\cdot\mathbf{x} + b)

    σ here is the sigmoid, not a standard deviation. The boundary p^=0.5\hat{p} = 0.5 is the line w⋅x+b=0\mathbf{w}\cdot\mathbf{x} + b = 0. The weights shown are the ones with the smallest mean log-loss on the training points.

  2. Press Train

    A network with one hidden layer of four units. Press Train and watch the boundary bend.

    The network has two inputs, four hidden units and one output. It starts from random weights. Each training step takes a batch of 20 points, works out which way to nudge every weight to lower the loss on them, and nudges. 10 steps make one epoch, one pass through all the training points. Training stops after 150 epochs.

    Two buttons, two jobs: Play steps through the lesson, and Train trains the network. Moving to another step pauses training.

    The shading is the network's p^\hat{p} for class 1 at every spot, and the solid curve is its boundary, where p^\hat{p} = 0.5. On the right, the training loss after each epoch, and the network's current loss and accuracy.

    On this seeded run the training loss falls from 0.71 to 0.054. Test accuracy first reaches 95% part-way through epoch 19, while the read-out still shows 18 finished epochs, and ends at 97.5%. Reset puts back the same starting weights, so pressing Train again repeats the run exactly.

    The first 20 epochs run at one step a frame so you can watch the bending; later epochs, where the boundary changes slowly, run a few steps a frame.

    The maths

    With layers numbered as on the notation page, the hidden layer and the output are

    a(1)=tanh⁡ ⁣(W(1)x+b(1)),p^=σ(w(2)⋅a(1)+b(2))\mathbf{a}^{(1)} = \tanh\!\big(\mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}\big), \qquad \hat{p} = \sigma\big(\mathbf{w}^{(2)}\cdot\mathbf{a}^{(1)} + b^{(2)}\big)

    Here W(1)\mathbf{W}^{(1)} is a 4 × 2 matrix, one row of weights per hidden unit, and tanh acts on each entry. σ here is the sigmoid, not a standard deviation. The loss L\mathcal{L} is binary cross-entropy, the log-loss of Level 3, averaged over a batch of 20 points:

    L=−120∑i∈batch[ yiln⁡p^i+(1−yi)ln⁡(1−p^i)]\mathcal{L} = -\frac{1}{20}\sum_{i \in \text{batch}} \Big[\, y_i \ln \hat{p}_i + (1 - y_i)\ln(1 - \hat{p}_i) \Big]

    Each step moves every weight a little against its gradient, as in gradient descent. The steps use Adam with learning rate η = 0.03, a refinement of gradient descent met later in this level. How the gradients are found is the next lesson.

  3. Each hidden unit draws a line

    Each hidden unit has its own straight line. The output unit adds up which side of each line a point is on.

    The dashed lines are the hidden units' lines, one per unit: where its weighted sum wⱼ·x + bⱼ is zero. Across its line a unit's output tanh switches from near −1 on one side to near 1 on the other.

    The output unit takes a weighted sum of those four switches. A point inside all four lines gets one total and a point outside some of them another, so the boundary can follow the lines and turn at their corners. The tanh switch is smooth, which rounds the corners off.

    Press Train (or carry on from step 2: it is the same network) and watch the lines move. The diagram shows each weight: thicker for larger, solid for positive, dashed for negative. By the end of the run all four lines sit between 1.0 and 1.4 from the centre, close around the inner ring. Nobody placed them there; the weights moved there because it lowered the loss.

    The maths

    Hidden unit jj computes aj=tanh⁡(wj⋅x+bj)a_j = \tanh(\mathbf{w}_j\cdot\mathbf{x} + b_j), where wj\mathbf{w}_j is row jj of W(1)\mathbf{W}^{(1)}. Its dashed line is wj⋅x+bj=0\mathbf{w}_j\cdot\mathbf{x} + b_j = 0. The output's logit is

    z(2)=∑j=14wj(2)tanh⁡(wj⋅x+bj)+b(2)z^{(2)} = \sum_{j=1}^{4} w^{(2)}_j \tanh(\mathbf{w}_j\cdot\mathbf{x} + b_j) + b^{(2)}

    a sum of bent functions of x\mathbf{x}, so the set where it is zero can be curved. Without the tanh it would be one straight line again: if the hidden layer were linear,

    w(2)⋅(W(1)x+b(1))+b(2)=(W(1)Tw(2))⋅x+(w(2)⋅b(1)+b(2))\mathbf{w}^{(2)}\cdot\big(\mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)}\big) + b^{(2)} = \big(\mathbf{W}^{(1)\mathsf{T}}\mathbf{w}^{(2)}\big)\cdot\mathbf{x} + \big(\mathbf{w}^{(2)}\cdot\mathbf{b}^{(1)} + b^{(2)}\big)

    which is one weight vector and one bias: a single neuron. A product of matrices is one matrix, so stacking linear layers never adds anything. The bend comes from the activation.

  4. How many hidden units?

    Choose the number of hidden units H and press Train. Each choice starts from the same seeds, and the table fills in as each run finishes.

    Every H uses the same data, the same seeds and the same settings; only the number of hidden units changes. When a run reaches 150 epochs its final training and test accuracy go into the table.

    On these runs, with 1 hidden unit the network ends at 63.2% test accuracy: one unit means one line. With 2 units it ends at 79.5%: two lines cannot close a ring. With 3 units it reaches 97.3%, with 4 97.5% and with 8 98.8%.

    Three lines are the fewest that can fence in the inner ring, and three units are the fewest that succeed here. That fits the picture, but it is what these seeded runs show, not a rule: other data, seeds or settings give other numbers.

    The maths

    There is a theorem about what enough hidden units can do. Stated with its conditions: a network with one hidden layer and enough units can approximate any continuous function on a bounded region as closely as you like. It does not say how many units are needed, or that training will find the weights.

    The final accuracies on the training points are 69.5%, 86.5%, 97.5%, 99.5% and 99.5% for H = 1, 2, 3, 4 and 8. They are higher than the test accuracies because the weights were tuned on these points.

  5. What the hidden layer sees

    With two hidden units, draw each point at its two hidden outputs (a₁, a₂). The output unit only sees this picture, and in it the boundary is a straight line.

    Each training point is plotted at the outputs of the two hidden units, a₁ across and a₂ up, both between −1 and 1. The output unit is one neuron reading these two numbers, so its boundary here is a straight line, drawn solid.

    Press Train and watch the points move. The hidden layer bends and squeezes the plane so that a straight line separates the classes better. At the end of the run that line gets 86.5% of the training points right, against 57.0% for the one-neuron fit of step 1 and 71.5% for the best possible line on the original x₁ and x₂.

    That is exactly the network's accuracy, not by luck: the line here is the network's last step, so the two are the same calculation. Two units cannot pull the rings fully apart, which is why H = 2 fell short in step 4.

    The maths

    The output unit computes p^=σ(w1(2)a1+w2(2)a2+b(2))\hat{p} = \sigma(w^{(2)}_1 a_1 + w^{(2)}_2 a_2 + b^{(2)}), so in the (a1,a2)(a_1, a_2) plane its boundary is the line w1(2)a1+w2(2)a2+b(2)=0w^{(2)}_1 a_1 + w^{(2)}_2 a_2 + b^{(2)} = 0. A network with one hidden layer is a mapping x↦a(1)\mathbf{x} \mapsto \mathbf{a}^{(1)} followed by a logistic regression on a(1)\mathbf{a}^{(1)}. Training tunes both at once.

  6. What the network learned, and what it didn't

    Training lowered the loss on the training points. Away from them, the boundary is whatever the weights happen to give.

    This is the four-unit network from step 2, zoomed out to 8 in every direction. The numbers in this text are for the end of its 150-epoch run; if the read-out does not say finished, press Train. The dashed square is the area of the earlier steps; no point lies further than 2.9 from the centre.

    On the dotted circle of radius 7, where there are no data at all, the trained network gives p^\hat{p} of at least 0.95 in every direction: confident, with nothing to go on. That answer comes from where its four lines happen to sit, not from anything in the data.

    Near the data, the boundary is not the circle you might draw by eye: along different directions it crosses between 1.3 and 1.8 from the centre. The network found weights that lower the training loss; it was never told the classes are rings.

    "Learns" in this level means exactly that: adjusts its weights to lower the loss on the training data. The final training accuracy, 99.5%, is above the test accuracy, 97.5%. With many more units than the data need, that gap can grow into overfitting, which is why a test set is kept aside.

    The maths

    Far from the centre each tanh⁡(wj⋅x+bj)\tanh(\mathbf{w}_j\cdot\mathbf{x} + b_j) is close to ±1\pm 1, set by the side of line jj the point is on. So far out, p^\hat{p} is nearly constant within each region the four lines mark off, and its value there was never checked against any data.

    The shading is worked out on a grid of 54 × 50 squares, each coloured by p^\hat{p} at its centre; the boundary curve is traced through that grid where p^\hat{p} = 0.5.

Quick check

Why does the network need a non-linear activation?

Right. A linear layer followed by a linear layer is one linear layer, because a product of matrices is one matrix. The bend in the boundary comes from the activation.

Speed is not the reason. Without the activation the network can only draw one straight line, however long it trains.

Here tanh gives values between −1 and 1, which are not probabilities. Only the output's sigmoid gives p^\hat{p}. The point of tanh is the bend.

The name comes from a loose analogy, but the reason is mathematical: without a non-linear activation the layers collapse into one straight-line score.

With 2 hidden units, what does the output neuron see?

Right. The output unit is one neuron reading (a₁, a₂), so in that plane its boundary is a straight line. Step 5 draws exactly that.

Only the hidden units read x₁ and x₂. The output reads their outputs, and in that plane its boundary is straight; it looks curved only when drawn back over x₁ and x₂.

The network handles one point at a time: for each point the output reads that point's two hidden outputs. Training uses batches, but each prediction is for one point.

It sees numbers, not pictures: each hidden unit's output, which says how far and on which side of that unit's line the point lies, squashed by tanh.

Lesson complete

  • Each hidden unit draws a straight line; the output unit combines them, so the network's boundary can bend.
  • Training adjusts every weight, step by step, to lower the loss on the training data; on these rings, three, four or eight hidden units bent the boundary round the inner ring.
  • The non-linear activation is what makes the bend possible: without it the layers collapse into one straight line.

Common misconceptions

  • "Adding hidden units can only help." On these rings more units never hurt the test accuracy, but that is these data and these runs. More units also give more room to fit noise in the training data, which is overfitting.
  • "The network has worked out that the data are rings." Its boundary is not a circle, and far from the data it answers confidently with nothing to go on. It found weights that lower the training loss.
  • "A network with one hidden layer can fit any shape, so training will find it." The theorem says enough units can approximate any continuous function on a bounded region; it does not say how many, or that training finds those weights.
Next lesson: Backpropagation →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 6, "Deep Feedforward Networks", which covers hidden layers, activations and the universal approximation theorem.
  2. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), its chapter on deep learning, which starts from a network with a single hidden layer.

About the data. All the data are made up, in your browser, by the mulberry32 random number generator. Each point gets an angle θ drawn uniformly from 0 to 2π and a radius r = r₀ + 0.25 z, where z is a draw from a normal distribution with mean 0 and SD 1; the point is (r cos θ, r sin θ). Class 0 has r₀ = 1 and class 1 has r₀ = 2.2, half the points each, class 0 first. The 200 training points use seed 151 and the 400 test points seed 152. The network's starting weights are drawn with seed 155 (normal, variance 1 divided by the number of inputs to the layer; biases start at 0), and the order of the training points in each epoch is shuffled with seed 154. Every run uses Adam with η = 0.03, batches of 20 and 150 epochs.

If this helped, you can .