Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 4 · Deep learning

The artificial neuron

A weighted sum, a bias and a squashing curve: the building block of every neural network, and an old friend in disguise.

About 10 minutes · No maths needed; the maths is there if you want it

Overview

Neural networks are built from one small part, repeated many times: the artificial neuron. This lesson takes one neuron apart. It adds up its inputs with weights, adds a bias, and passes the total through a curve called an activation.

With the sigmoid as its activation, one neuron is exactly the logistic regression of Level 3. That tells us what it can do, and what it cannot: its boundary is a straight line. On data arranged like a two-by-two chessboard, no straight line will do, and that is why networks have more than one neuron.

The data are made up. See "About the data" at the end.

You'll learn

  • what a neuron computes: a weighted sum plus a bias, passed through an activation
  • why a single sigmoid neuron is logistic regression, so its boundary is a straight line
  • why one neuron cannot split XOR data, and how two lines can
  1. Inputs, weights, sum

    A neuron multiplies each input by a weight, adds them up and adds a bias. The total is called z.

    Each point has two features, x₁ across and x₂ up. There are 60 points: cyan circles are class 0 and violet triangles class 1.

    The diagram on the right is one neuron. The two inputs on the left are the ringed point's features, x₁ = 1.14 and x₂ = 0.63. Each link carries a weight, w₁ or w₂, and the neuron adds a bias b. The number in the neuron is

    z = w₁x₁ + w₂x₂ + b.

    With the starting weights w₁ = 1.00, w₂ = −0.50 and b = 0.50, the ringed point gets z = 1.32. Move the sliders and watch z change. A thicker link means a larger weight; a dashed violet link means a negative one.

    The name comes from a loose analogy with nerve cells, which add up signals and fire; none of the maths here depends on that analogy.

    The maths

    Collect the inputs into a vector x=(x1,x2)\mathbf{x} = (x_1, x_2) and the weights into w=(w1,w2)\mathbf{w} = (w_1, w_2). The weighted sum is a dot product:

    z=w⋅x+b=w1x1+w2x2+bz = \mathbf{w}\cdot\mathbf{x} + b = w_1 x_1 + w_2 x_2 + b

    This is the same straight-line score as logistic regression and linear regression with many features. The bias bb plays the part of the intercept: it shifts zz up or down for every point at once.

  2. Activation

    The neuron passes z through a curve g, its activation. Its output is a = g(z).

    Pick an activation. The plot shows its curve, with the ringed point's z marked across and its output a marked up.

    At the starting z = 1.32: the step gives 1.00, the sigmoid 0.79, tanh 0.87 and ReLU 1.32.

    • Step: 1 if z > 0, otherwise 0. A hard yes or no.
    • Sigmoid σ: a smooth S from 0 to 1. σ here is the sigmoid, not a standard deviation.
    • tanh: the same S shape, stretched to run from −1 to 1.
    • ReLU: z itself when z is positive, otherwise 0.

    The read-out also shows the slope of the curve at z. The step's slope is 0 everywhere except at the jump, where it has none. That matters later: training adjusts the weights by following slopes, so the step gives it nothing to follow.

    The maths
    a=g(z),σ(z)=11+e−z,tanh⁡z=ez−e−zez+e−z,ReLU⁡(z)=max⁡(0,z)a = g(z), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \tanh z = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}, \qquad \operatorname{ReLU}(z) = \max(0, z)

    σ here is the sigmoid, not a standard deviation. Their slopes have neat forms:

    σ′(z)=σ(z)(1−σ(z)),tanh⁡′(z)=1−tanh⁡2z,ReLU⁡′(z)={1z>00z<0\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big), \qquad \tanh'(z) = 1 - \tanh^2 z, \qquad \operatorname{ReLU}'(z) = \begin{cases} 1 & z > 0 \\ 0 & z < 0 \end{cases}

    ReLU has no slope at exactly z=0z = 0; the page uses 0 there, a common convention. The two S curves are related: tanh⁡z=2σ(2z)−1\tanh z = 2\sigma(2z) - 1.

  3. It is logistic regression

    With a sigmoid, the neuron's output is σ(w₁x₁ + w₂x₂ + b): exactly the p^\hat{p} of logistic regression.

    From here on the neuron uses the sigmoid. Its output is a number between 0 and 1, which we read as p^\hat{p}, the probability of class 1. That is the model of logistic regression, written as a diagram.

    Press Fit. The sliders jump to the weights logistic regression finds for these data: w₁ = 2.57, w₂ = 3.48 and b = 1.13. They are the weights with the smallest log-loss.

    The solid line is where p^\hat{p} = 0.5. With the fitted weights, predicting class 1 on one side and class 0 on the other gets 95.0% of the points right.

    With those weights, the neuron's output and logistic regression's p^\hat{p} agree for every point to within 10⁻¹⁵, the size of rounding error. They are the same calculation.

    The maths
    a=σ(w⋅x+b)=p^a = \sigma(\mathbf{w}\cdot\mathbf{x} + b) = \hat{p}

    σ here is the sigmoid, not a standard deviation. Fitting chooses w\mathbf{w} and bb to make the mean log-loss as small as possible. In deep learning the same loss is called binary cross-entropy:

    L=−1n∑i=1n[ yiln⁡ai+(1−yi)ln⁡(1−ai)]\mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n} \Big[\, y_i \ln a_i + (1 - y_i)\ln(1 - a_i) \Big]

    Here L\mathcal{L} is the mean over all nn points. The Fit button uses Newton's method, as in Level 3. Gradient descent would approach the same weights, because this loss is convex.

  4. One neuron, one straight line

    Whatever the weights, the neuron splits the plane with a straight line.

    The shading shows the neuron's output σ(z) at every spot on the plot: the stronger the violet, the closer to 1; the stronger the cyan, the closer to 0. The solid line is the boundary, where z = 0.

    Move the sliders. The line turns and slides, and the shading follows, but the line never bends. The weights w₁ and w₂ set its direction and b shifts it. With w₁ = w₂ = 0 there is no line at all: z = b everywhere.

    None of the four activations on this page changes this. The output depends on the point only through z, so all points with the same z get the same output, and those points lie on a straight line. All four only rise or stay flat as z rises, so the side of the line with z > 0 always gets the higher output, and one straight boundary is all there is.

    The maths

    The boundary is the set

    { x:w⋅x+b=0 }\{\, \mathbf{x} : \mathbf{w}\cdot\mathbf{x} + b = 0 \,\}

    which, for w≠0\mathbf{w} \neq \mathbf{0}, is a straight line at right angles to w\mathbf{w}. Every level set w⋅x+b=c\mathbf{w}\cdot\mathbf{x} + b = c is a line parallel to it, so the output is constant along lines parallel to the boundary.

    The shading is worked out on a grid of 54 × 50 small squares, each coloured by σ(z) at its centre. The line is drawn exactly.

  5. XOR: where one line fails

    Put class 1 in two opposite corners and class 0 in the other two. No straight line can split them.

    New made-up data: 80 points in four groups around the corners (±1, ±1). A point is class 1 when its corner's two signs differ, top left and bottom right, and class 0 otherwise. This pattern is called XOR, "exclusive or".

    Move the sliders and try to split them. Whatever line you find, some group ends up on the wrong side.

    The page checks every line it can: 360 directions, and for each direction every place the line could cut between points, with class 1 on either side. The best of all those lines gets 77.5% right. A second check, which tries every way a straight line can split these 80 points, finds the same 77.5%.

    For scale, 42 of the 80 points are class 1, so calling everything class 1 already gets 52.5%. The best line does better than that, but it still gets 18 of the 80 wrong, whatever the weights.

    The maths

    Take the four corners alone. A line would need z>0z > 0 at (1,−1)(1, -1) and (−1,1)(-1, 1) and z<0z < 0 at (1,1)(1, 1) and (−1,−1)(-1, -1):

    w1−w2+b>0,−w1+w2+b>0,w1+w2+b<0,−w1−w2+b<0w_1 - w_2 + b > 0, \quad -w_1 + w_2 + b > 0, \quad w_1 + w_2 + b < 0, \quad -w_1 - w_2 + b < 0

    Adding the first two gives 2b>02b > 0; adding the last two gives 2b<02b < 0. Both cannot hold, so no weights work. The noise around each corner does not rescue it.

    Why the second check is complete: the order of the points along a direction only changes at directions where two points line up, so trying one direction between each such pair covers every possible split.

  6. Where next: two neurons, two lines

    Two neurons can draw two lines, and a third neuron can combine them. That is the idea of the next lesson.

    Here three step neurons are wired by hand; nothing is trained. The first fires when x₁ − x₂ > 1, which is everything below the dashed line 1. The second fires when x₂ − x₁ > 1, above the dashed line 2. The output neuron fires when either of them does.

    The band between the lines is class 0 and the two outer regions are class 1. This network gets 98.8% of the points right, against 77.5% for the best single line.

    The two neurons in the middle are called a hidden layer. In the next lesson a network with a hidden layer learns where to put its lines: it adjusts its weights to lower the loss on the training data, and the boundary bends as it does.

    The maths

    With the step function as gg, the hand-set network is

    h1=g(x1−x2−1),h2=g(−x1+x2−1),output=g(h1+h2−0.5)h_1 = g(x_1 - x_2 - 1), \qquad h_2 = g(-x_1 + x_2 - 1), \qquad \text{output} = g(h_1 + h_2 - 0.5)

    The two hidden neurons have weights (1,−1)(1, -1) and (−1,1)(-1, 1), both with bias −1-1; the output neuron has weights (1,1)(1, 1) and bias −0.5-0.5. The diagram shows them: solid links positive, dashed negative.

Quick check

What does a single sigmoid neuron compute?

Right. a = σ(w·x + b) is the logistic regression model. With the same weights the two agree to within rounding error.

The sigmoid bends the output, not the boundary. The boundary is where z = 0, and z is a straight-line score, so the boundary is a straight line.

The weighted sum is z. The sigmoid then squashes it into a number between 0 and 1.

The name comes from a loose analogy, but the neuron is a short formula: a weighted sum, a bias and an activation.

Why can no single neuron solve XOR?

Right. On these data the best line of all gets 77.5% right. The corners alone already rule a line out: the conditions for the four corners contradict each other.

The output depends on the point only through z, and ReLU, like the sigmoid, only rises or stays flat as z rises. With any of the four activations here, the neuron's regions are split by one straight line.

More points in the same four groups would not help. Even the four corners on their own cannot be split by a line.

The page checked every possible line, not just one found by training. None does better than 77.5%.

Lesson complete

  • A neuron computes z = w·x + b and outputs a = g(z), where g is its activation: step, sigmoid, tanh or ReLU.
  • With a sigmoid, one neuron is exactly logistic regression, and fitting it means making the log-loss (binary cross-entropy) small.
  • With any of the four activations here (all only rise or stay flat as z rises), one neuron's boundary is a straight line, so it cannot split XOR data.
  • Two neurons can draw two lines and a third can combine them. Learning where to put the lines is the next lesson.

Common misconceptions

  • "An artificial neuron works like a brain cell." The name is a loose analogy. The neuron is a weighted sum, a bias and an activation.
  • "A different activation would let one neuron bend the boundary." The output depends on the point only through z, and step, sigmoid, tanh and ReLU only rise or stay flat as z rises, so with any of them the boundary stays one straight line.
  • "The sigmoid makes the boundary curved." It curves the output along the direction of w; the boundary z = 0 is still a line.
Next lesson: Hidden layers: bending the boundary →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Chapter 6, "Deep Feedforward Networks", which builds networks from units like this one.
  2. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), its chapter on deep learning, which starts from a network with a single hidden layer.

About the data. All the data are made up, in your browser, by the mulberry32 random number generator. The 60 points in steps 1 to 4 use seed 143: x₁ = m₁ + 0.8 z₁ and x₂ = m₂ + 0.8 z₂, where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1. Class 0 has centre (m₁, m₂) = (−1, −1) and class 1 has centre (1.2, 1), with 30 points each, class 0 first; the ringed point is the second point of class 1. The 80 XOR points in steps 5 and 6 use seed 142: for each point a corner (±1, ±1) is chosen at random with equal chances, then x₁ and x₂ are the corner's coordinates plus 0.35 times independent normal draws with mean 0 and SD 1. A point is class 1 when the corner's two signs differ.

If this helped, you can .