Skip to the content
ML Explorer Reju Sam John

ML Explorer · Reference

Notation

Every symbol the lessons use, what it means, and the lesson that introduces it.

The lessons keep one letter for one idea wherever they can. Where a lesson uses a letter differently, the entry says so in a note.

Most examples use the same three points, (x,y)(x, y) = (1, 1), (2, 3) and (3, 2). Their least-squares line is y^=1+0.5 x\hat{y} = 1 + 0.5\,x.

Data and examples

What the lessons learn from: inputs, answers and the misses in between.

xx

An input value, also called a feature: the thing the model is given.

ExampleIn Lesson 1, xx is the hours a student studied.

Introduced in Lesson 1.

yy

The answer that goes with an input, also called the target or label: the thing the model should predict. It is never one of the features.

ExampleIn Lesson 1, yy is the exam mark.

Introduced in Lesson 1.

y∈{0,1}y \in \{0, 1\}

A class label. In Level 3 the answer is often a class rather than an amount, and the two classes are called 0 and 1. These are names: class 1 is not “more” than class 0. On the plots, class 0 is a cyan circle and class 1 a violet triangle.

Exampleyi∈{0,1}y_i \in \{0, 1\} says that case ii belongs to class 0 or class 1.

NoteLesson 14 writes the classes as yi=±1y_i = \pm 1 (that is, y∈{−1,+1}y \in \{-1, +1\}), so that one inequality covers both sides of the line. Lesson 18’s data have no labels at all.

Introduced in Lesson 11.

y^\hat{y}

A prediction of yy, said “y-hat”. A hat marks a value worked out from data rather than observed.

Exampley^2=1+0.5×2=2\hat{y}_2 = 1 + 0.5 \times 2 = 2

NoteFor a classifier, y^\hat{y} is a predicted class, 0 or 1 (Lesson 11 on). Lesson 19’s x^i\hat{\mathbf{x}}_i is a point rebuilt from a single number.

Introduced in Lesson 1.

nn, ii

nn is the number of examples. ii numbers them, from 1 to nn.

ExampleFor the three points, n=3n = 3.

Introduced in Lesson 1.

xix_i, yiy_i

The input and the answer of example ii.

Examplex2=2, y2=3x_2 = 2,\ y_2 = 3

Introduced in Lesson 1.

ei=yi−y^ie_i = y_i - \hat{y}_i

The residual of example ii: how far the answer is from the prediction. It is positive when the point lies above the line.

Examplee2=3−2=1e_2 = 3 - 2 = 1

NoteLesson 17 writes rmir_{mi} for the residual after m−1m - 1 rounds of boosting. For class labels a difference yi−y^iy_i - \hat{y}_i has no useful meaning (Lesson 11).

Introduced in Lesson 1; Lesson 4 builds losses from it.

ε\varepsilon

Noise: everything the rule does not capture. In the lessons’ made-up data it is the random part added to each yy.

Exampley=f(x; θ)+εy = f(x;\,\boldsymbol{\theta}) + \varepsilon

NoteLesson 4 and Lesson 9 write εi\varepsilon_i for the noise on example ii.

Introduced in Lesson 1.

dd

The number of features each example has.

ExampleIn Lesson 2, n=6n = 6 examples have d=2d = 2 features each.

NoteThe maths of Lesson 8 writes pp for the number of features. In Lesson 10, pp is the degree of a polynomial instead. Lesson 13 writes dd for the number of features, as here, but in Lesson 15 dd is the depth of a tree.

Introduced in Lesson 2.

x\mathbf{x}

A vector: a list of numbers written in bold. It is also a point you can draw.

Examplex=(3,4)\mathbf{x} = (3, 4)

NoteLesson 2 also names two points a\mathbf{a} and b\mathbf{b}. Those bold letters are points, not the intercept and slope aa and bb.

Introduced in Lesson 2.

xi\mathbf{x}_i

All the features of example ii, as one vector.

Examplexi=(xi1, xi2)\mathbf{x}_i = (x_{i1},\, x_{i2})

Introduced in Lesson 2.

xijx_{ij}

The value of feature jj for example ii. With two subscripts, the first is the example and the second the feature.

Examplex21x_{21} is example 2’s value of feature 1.

Introduced in Lesson 2.

x1,…,xdx_1, \dots, x_d

Inside one vector, a single subscript numbers its entries: the features of one example.

Examplex=(x1,…,xd)\mathbf{x} = (x_1, \dots, x_d). In Lesson 8’s maths, y^=a+w1x1+⋯+wpxp\hat{y} = a + w_1 x_1 + \dots + w_p x_p.

NoteSo x2x_2 can mean example 2’s input (as in xix_i) or feature 2 of one vector; the text says which. In Lesson 5, (x1,y1)(x_1, y_1) and (x2,y2)(x_2, y_2) are two points on a line.

Introduced in Lesson 2.

R2\mathbb{R}^2

The set of all pairs of real numbers: every point of a flat plane. More generally, Rd\mathbb{R}^d holds every list of dd real numbers.

Examplexi=(xi1, xi2)∈R2\mathbf{x}_i = (x_{i1},\, x_{i2}) \in \mathbb{R}^2: each example is a point in the plane.

Introduced in Lesson 2.

X\mathbf{X}, XX

The data matrix: one row per example and one column per feature, so nn rows and dd columns.

NoteLesson 2 writes it in bold, X\mathbf{X}. The maths of Lesson 8 and Lesson 10 writes XX and puts a column of 1s first, for the intercept. In Lesson 10 the other columns are powers of xx.

Introduced in Lesson 2.

y\mathbf{y}

All the answers in one vector, y=(y1,…,yn)\mathbf{y} = (y_1, \dots, y_n).

Exampley=(1,3,2)\mathbf{y} = (1, 3, 2) for the three points.

Used in the maths of Lesson 8 and Lesson 10.

BB, nBn_B, EE

A batch BB is the small set of training points that one training step uses, and nBn_B is how many points it holds. One pass through all nn training points is an epoch, and EE is the number of epochs. Each epoch the points are shuffled and cut into n/nBn/n_B batches.

ExampleWith n=160n = 160 points in batches of nB=16n_B = 16, an epoch is n/nB=160/16=10n/n_B = 160/16 = 10 steps.

NoteLesson 16’s BB is the number of trees, and Lesson 13’s nBn_B counts the cases in a band. Pages that need no symbol give the batch size in words.

Introduced in Lesson 23.

X[r,c]X[r, c]

An image as a grid of numbers: X[r,c]X[r, c] is the pixel in row rr and column cc, both counted from 0 at the top left.

ExampleLesson 26’s images are 12 × 12, so r,c=0,1,…,11r, c = 0, 1, \ldots, 11. Lesson 27’s are 8 × 8.

NoteAn image is not the data matrix XX, which holds one example per row; in Lesson 27 each image is one example.

Introduced in Lesson 26.

xtx_t, tt, TT

A sequence of inputs x1,…,xTx_1, \dots, x_T, one for each time step t=1,…,Tt = 1, \dots, T. A recurrent network reads them in order.

ExampleIn Lesson 28 each sequence holds T=12T = 12 bits.

NoteThe subscript tt numbers time steps, where xix_i numbers examples. TT is a number of steps here, not the test set of Lesson 9 and not a transpose.

Introduced in Lesson 28.

V\mathcal{V}, ∣V∣|\mathcal{V}|, ei\mathbf{e}_i

The vocabulary V\mathcal{V}, a curly V: the fixed list of every token a model can use. ∣V∣|\mathcal{V}| is its size, and the tokens have ids 0,1,…,∣V∣−10, 1, \dots, |\mathcal{V}| - 1. An id is a name, not an amount, so a model turns id ii into the one-hot vector ei\mathbf{e}_i: ∣V∣|\mathcal{V}| numbers, all 0 except a 1 in entry ii, counting entries from 0 like the ids. A text becomes a sequence of ids x1,x2,…x_1, x_2, \dots, one for each position tt.

ExampleIn Lesson 29, the vocabulary is space, full stop and the 26 letters, so ∣V∣=28|\mathcal{V}| = 28. The letter a has id 2, so it becomes e2=(0,0,1,0,…,0)\mathbf{e}_2 = (0, 0, 1, 0, \dots, 0).

NoteLesson 30’s vocabulary is 27 words and Lesson 33’s the 10 digits. Each merge in Lesson 29 adds one token to V\mathcal{V}. The curly V\mathcal{V} is never the bold V\mathbf{V} of attention. Here the subscript tt numbers positions in a text, as it numbers time steps in Lesson 28.

Introduced in Lesson 29; used throughout Level 5.

Summaries of a set of values

Centre and spread, the two numbers that sum up a set of values. The examples use the values x=1,2,3x = 1, 2, 3 throughout.

xˉ\bar{x}, yˉ\bar{y}

The mean of the values, said “x-bar”: add them up and divide by nn. yˉ\bar{y} is the mean of the yy values.

Examplexˉ=(1+2+3)/3=2\bar{x} = (1 + 2 + 3)/3 = 2

NoteIn bold, xˉ\bar{\mathbf{x}} is the mean point of a set of vectors, worked out one feature at a time (Lesson 18 and Lesson 19). Lesson 18’s xˉk\bar{\mathbf{x}}_k is the mean of the points in cluster kk.

Introduced in Lesson 3. It appears earlier in the maths of Lesson 1.

μ\mu

The population mean, said “mu”: the mean of the whole group you care about. When the values are the whole group, μ=xˉ\mu = \bar{x}.

Exampleμj\mu_j is the mean of feature jj.

NoteIn Lesson 3’s maths, μ\mu is also the unknown mean of a larger population that a sample was drawn from. In Lesson 8 it is the mean of xx. Lesson 13’s μc\mu_c and μcj\mu_{cj} are the means of one class. Lesson 18’s bold μk\boldsymbol{\mu}_k is a cluster centre, and Lesson 35’s the mean of a mixture component. In Lesson 33, μ\mu is the mean of one position’s numbers in layer normalisation.

Introduced in Lesson 2 (as μj\mu_j) and Lesson 3.

σ\sigma, σ2\sigma^2

The population standard deviation and variance, said “sigma”. The variance is the mean squared distance from the mean, dividing by nn. The SD is its square root, in the units of the data.

Exampleσ2=2/3≈0.67\sigma^2 = 2/3 \approx 0.67, so σ≈0.82\sigma \approx 0.82

NoteLesson 7 writes σx2\sigma_x^2 for the population variance of xx. In Lesson 10, σ2\sigma^2 is the variance of the noise ε\varepsilon, not of the data. In Lesson 3’s maths, σ2\sigma^2 is also the unknown variance of a larger population. Lesson 13’s σc2\sigma_c^2 and σcj2\sigma_{cj}^2 are the variances of one class, dividing by the class size ncn_c. Lesson 16’s σ2\sigma^2 is how much one tree’s prediction varies from one training set to another. Written with a bracket, σ(z)\sigma(z) is the sigmoid of Lesson 12, not a standard deviation. Lesson 36’s σt\sigma_t is the standard deviation of the noise added at step tt.

Introduced in Lesson 2 (as σj\sigma_j) and Lesson 3.

ss, s2s^2

The sample standard deviation and variance: the same sum of squared distances, divided by n−1n - 1 instead. They estimate the spread of the bigger group the values were drawn from.

Examples2=2/2=1s^2 = 2/2 = 1, so s=1s = 1

Introduced in Lesson 3. Lesson 19 writes s12s_1^2 and s22s_2^2 for the sample variances of two features.

SS, s12s_{12}

The sample covariance matrix of the features, dividing by n−1n - 1: S=1n−1X~⊤X~S = \frac{1}{n-1}\tilde{X}^{\top}\tilde{X}, with the centred data X~\tilde{X}. Its diagonal holds each feature’s sample variance, and s12s_{12}, off the diagonal, is the sample covariance of features 1 and 2: positive when they tend to rise together.

ExampleFor the three points, S=(10.50.51)S = \begin{pmatrix} 1 & 0.5 \\ 0.5 & 1 \end{pmatrix}.

NoteMany books write the covariance matrix as Σ\Sigma. These lessons keep Σ\Sigma for the summation sign.

Introduced in Lesson 19.

zz

A z-score: how many standard deviations a value lies from the mean, z=(x−μ)/σz = (x - \mu)/\sigma. Turning every value into its z-score is called standardising.

ExampleFor x=3x = 3: z=(3−2)/0.82≈1.22z = (3 - 2)/0.82 \approx 1.22

NoteEvery lesson that standardises (Lesson 2, Lesson 3 and Lesson 8) divides by the population SD σ\sigma, so the result has mean 0 and SD 1. From Lesson 12 on, zz is also a score fed to the sigmoid, and in Level 4 z(ℓ)\mathbf{z}^{(\ell)} is a layer’s pre-activation; neither is a z-score.

Introduced in Lesson 2 (as zijz_{ij}) and Lesson 3.

x(k)x_{(k)}

The kk-th smallest value, once the values are sorted. The median is the middle one (or the mean of the middle two when nn is even).

ExampleWritten in the order 3, 1, 2, the values sort to x(1)=1, x(2)=2, x(3)=3x_{(1)} = 1,\ x_{(2)} = 2,\ x_{(3)} = 3, so the median is x(2)=2x_{(2)} = 2.

Introduced in Lesson 3.

E[ ⋅ ]\mathbb{E}[\,\cdot\,]

Expected value: the long-run average of a random quantity over many repeats of the random draw.

ExampleE[s2]=σ2\mathbb{E}[s^2] = \sigma^2: for independent draws, the sample variance equals the variance of the population on average.

NoteLesson 36’s E\mathbb{E} is the mean over a random data point, step and noise draw. The blackboard-bold E\mathbb{E} is not the embedding matrix E\mathbf{E} of Level 5.

Introduced in Lesson 3; used again in Lesson 10 and Lesson 36.

Var⁡( ⋅ )\operatorname{Var}(\,\cdot\,)

The variance of a random quantity: how much it varies from one repeat to the next.

ExampleVar⁡(f^(x0))\operatorname{Var}\bigl(\hat{f}(x_0)\bigr) is how much the fitted curves disagree at x0x_0.

Introduced in Lesson 10.

P(c∣x)P(c \mid \mathbf{x}), p(xj∣c)p(x_j \mid c)

A probability, said “the probability of cc given x\mathbf{x}”: how likely class cc is once the features x\mathbf{x} are known. P(c)P(c) on its own is the prior, the share of class cc before looking at the features. A small pp is a density, the height of a curve such as the normal curve. A density can be above 1; the area under it is a probability.

ExampleBayes’ rule: P(c∣x)=P(x∣c) P(c)P(x)P(c \mid x) = \frac{P(x \mid c)\,P(c)}{P(x)}

Introduced in Lesson 13.

N(μ,Σ)\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}), ∼\sim

The normal, or Gaussian, distribution with mean μ\boldsymbol{\mu} and covariance matrix Σ\boldsymbol{\Sigma}; in one dimension it is N(μ,σ2)\mathcal{N}(\mu, \sigma^2), the bell curve. N(x∣μ,Σ)\mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}, \boldsymbol{\Sigma}) is its density at x\mathbf{x}. The sign ∼\sim means “is drawn from”. With 0\mathbf{0} a vector of zeros and I\mathbf{I} the identity matrix, ε∼N(0,I)\boldsymbol{\varepsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) is a vector of independent standard normal numbers.

ExampleThe standard normal density at its centre is N(0∣0,1)=1/2π≈0.4\mathcal{N}(0 \mid 0, 1) = 1/\sqrt{2\pi} \approx 0.4.

NoteLesson 13’s normal curve is the one-dimensional case. Inside the density, π\pi is the constant 3.14…, not a mixture weight.

Introduced in Lesson 35; used again in Lesson 36.

Sample versus population variance

Both average the squared distances from the mean. They differ in what they divide by, because they answer different questions.

For x=1,2,3x = 1, 2, 3, the population variance is about 0.67 and the sample variance is 1. The gap shrinks as nn grows.

Which lessons use which. Lesson 3 uses the sample version, ss and s2s^2, and shows the two side by side. Lesson 19 uses it too: its covariance matrix SS divides by n−1n - 1. Every other variance or standard deviation in the lessons divides by nn:

Models and their parameters

The numbers a model learns, and the settings that steer the learning. Examples use the three points (1, 1), (2, 3) and (3, 2).

aa, bb

The intercept and slope of a straight line y^=a+b x\hat{y} = a + b\,x. The intercept aa is the prediction at x=0x = 0; the slope bb is how much the prediction changes when xx goes up by 1.

Exampley^=1+0.5 x\hat{y} = 1 + 0.5\,x has a=1a = 1 and b=0.5b = 0.5.

NoteLesson 8 keeps aa for the intercept but writes the slope as ww. From Lesson 12 on, bb is the intercept instead, as in b+w xb + w\,x. Lesson 17’s aa and bb are the two levels of a stump.

Introduced in Lesson 1; also in Lesson 4, Lesson 5 and Lesson 6.

a^\hat{a}, b^\hat{b}

The intercept and slope chosen by least squares: the line with the smallest total squared error.

Examplea^=1, b^=0.5\hat{a} = 1,\ \hat{b} = 0.5

Introduced in Lesson 1; Lesson 6 derives the formula.

w\mathbf{w}, bb

The weights and intercept of a straight boundary between two classes. The score is w⋅x+b\mathbf{w}\cdot\mathbf{x} + b, and the boundary is the line where it is 0. w\mathbf{w} points across the line, towards the side where the score is positive.

ExampleWith w=(1,2)\mathbf{w} = (1, 2) and b=−3b = -3, the point (1,1)(1, 1) scores 1+2−3=01 + 2 - 3 = 0, so it lies on the boundary.

NoteHere bb is the intercept, not the slope of y^=a+b x\hat{y} = a + b\,x. With one feature, Lesson 12 writes the score as b+w xb + w\,x.

Introduced in Lesson 12; used in Lesson 13 and Lesson 14.

σ(z)\sigma(z)

The sigmoid: σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}. It squashes any score zz into a number between 0 and 1, with σ(0)=0.5\sigma(0) = 0.5.

Exampleσ(2)≈0.88\sigma(2) \approx 0.88, and σ(−2)≈0.12\sigma(-2) \approx 0.12.

Noteσ\sigma here is the sigmoid, not a standard deviation. Each page that uses it says so.

Introduced in Lesson 12. In Level 4 it is one of the activations gg (Lesson 20).

p^\hat{p}

A predicted probability of class 1. Logistic regression gives p^=σ(b+w x)\hat{p} = \sigma(b + w\,x), and a threshold tt turns it into a class: y^=1\hat{y} = 1 when p^≥t\hat{p} \ge t.

ExampleWith b=−3b = -3, w=1w = 1 and x=5x = 5 hours, p^=σ(−3+1×5)=σ(2)≈0.88\hat{p} = \sigma(-3 + 1 \times 5) = \sigma(2) \approx 0.88.

Notep^/(1−p^)\hat{p}/(1 - \hat{p}) is the odds, and its log is the log-odds. In Lesson 15, a plain pp or pkp_k is the share of a class in a group of cases, not a prediction.

Introduced in Lesson 12.

SxxS_{xx}, SxyS_{xy}, SyyS_{yy}

Sums of squared distances from the means, and of products of those distances: Sxy=∑i(xi−xˉ)(yi−yˉ)S_{xy} = \sum_i (x_i - \bar{x})(y_i - \bar{y}). The least-squares slope is b^=Sxy/Sxx\hat{b} = S_{xy}/S_{xx}.

ExampleSxx=2, Sxy=1, Syy=2S_{xx} = 2,\ S_{xy} = 1,\ S_{yy} = 2, so b^=1/2\hat{b} = 1/2.

Introduced in Lesson 6; used again in Lesson 7.

θ\boldsymbol{\theta}

All of a model’s parameters in one vector, said “theta”. θ^\hat{\boldsymbol{\theta}} is the fitted value.

Exampleθ=(a,b)\boldsymbol{\theta} = (a, b) for a straight line.

NoteLesson 8 writes θ=(a,w)\boldsymbol{\theta} = (a, w) and calls the lowest point θ∗\boldsymbol{\theta}^*. In Lesson 19, a plain θ\theta is the angle of a line instead, and in Lesson 30 the angle between two vectors. Lesson 35 and Lesson 36 write a plain θ\theta for a model’s parameters.

Introduced in Lesson 1.

θj\theta_j

Parameter jj of a polynomial: the number that multiplies xjx^j. θ0\theta_0 is the intercept.

Exampley^=θ0+θ1x+θ2x2+θ3x3\hat{y} = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3

Introduced in Lesson 9; Lesson 10 goes up to θp\theta_p.

pp

The degree of a polynomial: its highest power of xx. A degree-pp polynomial has p+1p + 1 parameters.

ExampleDegree 3 is θ0+θ1x+θ2x2+θ3x3\theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3.

NoteThe maths of Lesson 8 uses pp for the number of features instead.

Introduced in Lesson 10.

ff, f^\hat{f}

A function. In Lesson 1, f(x; θ)f(x;\,\boldsymbol{\theta}) is the model’s rule, with the parameters after the semicolon. In Lesson 5, ff is any curve whose slope we study. In Lesson 10, ff is the true curve behind the data and f^\hat{f} a fitted one.

Exampley^=f(x; θ)=a+b x\hat{y} = f(x;\,\boldsymbol{\theta}) = a + b\,x

Introduced in Lesson 1.

fˉ(x0)\bar{f}(x_0)

The average of many fitted curves at one input x0x_0. It estimates E[f^(x0)]\mathbb{E}[\hat{f}(x_0)].

ExampleIn Lesson 10 it averages the M=200M = 200 fits.

Introduced in Lesson 10.

ww

A weight: the number that multiplies an input. In these lessons it plays the part of the slope.

ExampleIn Lesson 7, y^=w x~\hat{y} = w\,\tilde{x}. In Lesson 8, y^=a+w x\hat{y} = a + w\,x. With pp features, wjw_j is the weight on feature jj.

NoteLessons 1 and 6 call the slope bb. Lesson 7 fits centred data, and its best ww equals Lesson 6’s b^\hat{b}.

Introduced in Lesson 7.

x~\tilde{x}, y~\tilde{y}

A centred value, said “x-tilde”: the value minus the mean, x~i=xi−xˉ\tilde{x}_i = x_i - \bar{x}. Likewise y~i=yi−yˉ\tilde{y}_i = y_i - \bar{y}.

Examplex=1,2,3x = 1, 2, 3 gives x~=−1,0,1\tilde{x} = -1, 0, 1.

Introduced in Lesson 7.

w∗w^*

The value of ww at the bottom of the loss: the best one.

Examplew∗=b^w^* = \hat{b}

Introduced in Lesson 7. Lesson 8 writes θ∗\boldsymbol{\theta}^* for the best pair.

LL

The loss, written as a function of the parameters: one number that says how badly they fit. Gradient descent tries to make it small.

ExampleL(w)L(w) in Lesson 7, L(a,w)L(a, w) in Lesson 8.

NoteIn Lesson 12, L(b,w)L(b, w) is the mean log-loss. In Lesson 17, L(y,F)L(y, F) is the loss of one prediction FF.

Introduced in Lesson 7.

η\eta

The learning rate, said “eta”: a small positive number that sets the size of each gradient descent step.

ExampleFrom w=1w = 1 with slope L′(w)=4L'(w) = 4 and η=0.1\eta = 0.1: 1−0.1×4=0.61 - 0.1 \times 4 = 0.6.

NoteLesson 12 fits logistic regression with gradient descent, so η\eta is the step size there too. In Lesson 17, η\eta is the boosting learning rate: each new stump is multiplied by η\eta before it is added. It plays the same part, setting the size of each step.

Introduced in Lesson 7; used again in Lesson 8, Lesson 12, Lesson 17 and throughout Level 4, from Lesson 21 on.

kk

A count. In gradient descent, the step number: wkw_k is ww after kk steps. In kk-fold cross-validation, the number of parts the training data is split into. In k-nearest neighbours, the number of neighbours that vote.

Examplewk+1=wk−η L′(wk)w_{k+1} = w_k - \eta\,L'(w_k)

NoteIn Lesson 18’s maths, kk numbers the clusters, as in μk\boldsymbol{\mu}_k; in Lesson 19 it numbers the principal components, as in γk\gamma_k. The number of clusters is a capital KK.

Steps: Lesson 7 and Lesson 8. Folds: Lesson 9. Neighbours: Lesson 11.

Nk(x)N_k(\mathbf{x}), k^\hat{k}

The kk training cases nearest to the point x\mathbf{x}: its neighbours, which vote on its class. k^\hat{k} is the value of kk that was picked as best.

ExampleWith k=3k = 3, N3(x)N_3(\mathbf{x}) holds the three training cases closest to x\mathbf{x}, and the class most of them have wins.

Introduced in Lesson 11.

KK

The number of clusters in k-means, a capital letter. You choose it; k-means does not learn it.

ExampleK=3K = 3 asks k-means for three groups.

NoteIt is a different thing from the kk of k-nearest neighbours. In Lesson 14, K(xi,xj)K(\mathbf{x}_i, \mathbf{x}_j) is a kernel: the dot product of two points after a change of features, worked out straight from the original features.

Introduced in Lesson 18.

c(i)c(i), μk\boldsymbol{\mu}_k

c(i)c(i) is the cluster that point ii is assigned to, a number from 1 to KK. μk\boldsymbol{\mu}_k is the centre of cluster kk, a point in the same space as the data.

Examplec(i)=arg⁡min⁡k∥xi−μk∥2c(i) = \arg\min_{k} \lVert \mathbf{x}_i - \boldsymbol{\mu}_k \rVert^2: each point joins its nearest centre.

NoteHere μk\boldsymbol{\mu}_k is a centre, not the population mean of the whole data set. The cluster numbers are names, with no order.

Introduced in Lesson 18.

aza_z, wzw_z

The intercept and weight after standardising xx: y^=az+wz z\hat{y} = a_z + w_z\,z. They map back with w=wz/σw = w_z/\sigma and a=az−wz μ/σa = a_z - w_z\,\mu/\sigma.

Exampleaz=2, wz=0.5×0.82≈0.41a_z = 2,\ w_z = 0.5 \times 0.82 \approx 0.41

Introduced in Lesson 8.

HH

The Hessian: the matrix of second derivatives of the loss. It describes how the loss curves in every direction.

ExampleH=2n X⊤XH = \frac{2}{n}\,X^{\top} X

Introduced in Lesson 8.

II

The identity matrix: 1s on the diagonal and 0s elsewhere. Multiplying by it changes nothing.

Example2I=(2002)2I = \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix}

NoteLevel 5 writes it bold, I\mathbf{I}, as in N(0,I)\mathcal{N}(\mathbf{0}, \mathbf{I}).

Used in the maths of Lesson 8, and in Level 5.

γmax⁡\gamma_{\max}, γmin⁡\gamma_{\min}

The largest and smallest eigenvalues of the Hessian, said “gamma”: how sharply the loss bowl curves along its steepest and its flattest direction.

ExampleFor standardised data in Lesson 8, H=2IH = 2I, so γmax⁡=γmin⁡=2\gamma_{\max} = \gamma_{\min} = 2.

NoteMost books write these eigenvalues as λ\lambda. These lessons keep λ\lambda for the regularisation strength. Lesson 19 writes the eigenvalues of the covariance matrix SS as γ1≥γ2\gamma_1 \ge \gamma_2: the variances along the two principal components. For the three points, γ1=1.5, γ2=0.5\gamma_1 = 1.5,\ \gamma_2 = 0.5.

Introduced in Lesson 8; used again in Lesson 19.

κ\kappa

The condition number, said “kappa”: κ=γmax⁡/γmin⁡\kappa = \gamma_{\max}/\gamma_{\min}. It is 1 for a round bowl and large for a long, narrow valley.

Exampleγmax⁡=8, γmin⁡=2\gamma_{\max} = 8,\ \gamma_{\min} = 2 gives κ=4\kappa = 4.

Introduced in Lesson 8.

λ\lambda

The regularisation strength, said “lambda”: how much the penalty on large parameters counts. λ=0\lambda = 0 is plain least squares.

ExampleWith λ=0.1\lambda = 0.1 and θ1=2\theta_1 = 2, that parameter adds 0.1×22=0.40.1 \times 2^2 = 0.4 to the loss.

NoteIn Lesson 23, λ\lambda is the weight decay strength, in the same role: a penalty on large weights. With plain gradient descent, each step first shrinks every weight by the factor 1−ηλ1 - \eta\lambda; with η=0.1\eta = 0.1 and λ=0.01\lambda = 0.01, 1−ηλ=1−0.1×0.01=0.9991 - \eta\lambda = 1 - 0.1 \times 0.01 = 0.999.

Introduced in Lesson 10.

J(θ)J(\boldsymbol{\theta})

The penalised loss of ridge regression: the MSE plus λ\lambda times the sum of the squared parameters. The intercept θ0\theta_0 is left out of the penalty.

ExampleJ(θ)=1n∑i=1nei2+λ∑j=1pθj2J(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^{n} e_i^2 + \lambda \sum_{j=1}^{p} \theta_j^2

NoteLesson 23 adds weight decay to a network’s loss, J(θ)=L(θ)+λ2∑ℓ∥W(ℓ)∥2J(\boldsymbol{\theta}) = \mathcal{L}(\boldsymbol{\theta}) + \frac{\lambda}{2}\sum_{\ell}\big\lVert \mathbf{W}^{(\ell)} \big\rVert^2, which penalises the weights but not the biases.

Introduced in Lesson 10.

CC

The penalty in a soft-margin support vector machine: how much each point inside the street, or on the wrong side of the line, costs. A large CC allows few such points; a small one buys a wider street.

ExampleSteps 3 and 4 of Lesson 14 use C=1C = 1.

NoteLesson 18’s CkC_k is the set of points in cluster kk, not a penalty.

Introduced in Lesson 14.

αi\alpha_i

One number per training point in the second form of the support vector problem, the dual, with 0≤αi≤C0 \le \alpha_i \le C. The weights are w=∑iαi yi xi\mathbf{w} = \sum_i \alpha_i\, y_i\, \mathbf{x}_i, so only points with αi>0\alpha_i > 0, the support vectors, fix the line.

ExampleA point well away from the street has αi=0\alpha_i = 0, and moving it a little changes nothing.

Introduced in Lesson 14.

ξi\xi_i

The slack of point ii, said “xi”: how far it is past its own edge of the street, ξi=max⁡(0, 1−yi(w⋅xi+b))\xi_i = \max\big(0,\ 1 - y_i(\mathbf{w}\cdot\mathbf{x}_i + b)\big). This is also called the hinge loss.

ExampleA point on or outside its own edge has ξi=0\xi_i = 0; a point exactly on the line has ξi=1\xi_i = 1.

Introduced in Lesson 14.

BB, f^b\hat{f}_b, SbS_b

BB is the number of trees in a bagged ensemble or random forest. f^b(x)\hat{f}_b(\mathbf{x}) is the class that tree bb predicts at x\mathbf{x}, and SbS_b is the set of rows in that tree’s bootstrap sample.

ExampleRow ii’s out-of-bag vote uses only the trees with i∉Sbi \notin S_b. In a sample of n=3n = 3 rows drawn with replacement, a given row is left out with probability (1−1/3)3=8/27≈0.3(1 - 1/3)^3 = 8/27 \approx 0.3.

NoteHere bb numbers the trees; it is not an intercept.

Introduced in Lesson 16.

FmF_m, hmh_m

In gradient boosting, FmF_m is the model after mm rounds and hmh_m is the small tree, a stump, added in round mm. The start F0=yˉF_0 = \bar{y} is the mean, and each round adds η\eta times a new stump: Fm(x)=Fm−1(x)+η hm(x)F_m(x) = F_{m-1}(x) + \eta\, h_m(x).

ExampleFM(x)=yˉ+η∑m=1Mhm(x)F_M(x) = \bar{y} + \eta \sum_{m=1}^{M} h_m(x) after MM rounds.

Introduced in Lesson 17.

rmir_{mi}

The residual of training point ii at the start of round mm: rmi=yi−Fm−1(xi)r_{mi} = y_i - F_{m-1}(x_i). It is the residual eie_i of earlier lessons, with y^i=Fm−1(xi)\hat{y}_i = F_{m-1}(x_i). Round mm fits its stump to these.

ExampleFor the three points, F0=yˉ=2F_0 = \bar{y} = 2, so r1i=yi−F0(xi)=−1, 1, 0r_{1i} = y_i - F_0(x_i) = -1,\ 1,\ 0.

NoteIt is not the correlation rr. For the squared loss 12(y−F)2\tfrac{1}{2}(y - F)^2, it is also minus the gradient of the loss.

Introduced in Lesson 17.

v\mathbf{v}, tit_i

A direction of length 1, v=(cos⁡θ,sin⁡θ)\mathbf{v} = (\cos\theta, \sin\theta) for a line at angle θ\theta. ti=x~i⋅vt_i = \tilde{\mathbf{x}}_i \cdot \mathbf{v} is where centred point ii lands along it. v1\mathbf{v}_1 and v2\mathbf{v}_2 are the principal component directions.

ExampleFor the three points, v1=(1,1)/2\mathbf{v}_1 = (1, 1)/\sqrt{2}, at 45°.

Introduced in Lesson 19.

ℓ\ell, LL

Layers of a network are numbered ℓ=1,…,L\ell = 1, \dots, L, said “ell”. The hidden layers come first and layer LL is the output layer. The input counts as layer 0.

ExampleLesson 25’s network has L=11L = 11: ten hidden layers and the output layer.

NoteIn Lesson 15, ℓ\ell is a leaf of a tree. Earlier lessons write the loss as LL; Level 4 writes it L\mathcal{L}.

Introduced in Lesson 22; used again in Lesson 25.

W(ℓ)\mathbf{W}^{(\ell)}, b(ℓ)\mathbf{b}^{(\ell)}

The weight matrix and bias vector of layer ℓ\ell. Row jj of W(ℓ)\mathbf{W}^{(\ell)} holds the weights into unit jj of the layer, so it has one row per unit and one column per input. b(ℓ)\mathbf{b}^{(\ell)} holds one bias per unit. A layer of noutn_{\text{out}} units fed by ninn_{\text{in}} inputs has nout(nin+1)n_{\text{out}}(n_{\text{in}} + 1) weights and biases.

ExampleLesson 25’s first layer has 8 units and 2 inputs: 8(2+1)=248(2 + 1) = 24 weights and biases.

NoteBold capital W\mathbf{W} is a matrix; a plain ww is one weight. When the output layer has a single unit, its weights form one row, written w(2)\mathbf{w}^{(2)} in Lesson 21, and its bias is one number, b(2)b^{(2)}. In Lesson 26 and Lesson 27, bold W\mathbf{W} is a filter instead.

Introduced in Lesson 21.

z(ℓ)\mathbf{z}^{(\ell)}, a(ℓ)\mathbf{a}^{(\ell)}

Layer ℓ\ell first forms its pre-activation, z(ℓ)=W(ℓ)a(ℓ−1)+b(ℓ)\mathbf{z}^{(\ell)} = \mathbf{W}^{(\ell)}\mathbf{a}^{(\ell-1)} + \mathbf{b}^{(\ell)}, then applies the activation to each entry: a(ℓ)=g(z(ℓ))\mathbf{a}^{(\ell)} = g\big(\mathbf{z}^{(\ell)}\big). The input is layer 0, a(0)=x\mathbf{a}^{(0)} = \mathbf{x}.

ExampleLesson 22’s network, with sigmoid hidden units, at x=(1,−0.5)\mathbf{x} = (1, -0.5): z(1)=W(1)x+b(1)=(0.800, −0.900)\mathbf{z}^{(1)} = \mathbf{W}^{(1)}\mathbf{x} + \mathbf{b}^{(1)} = (0.800,\ -0.900) and a(1)=σ(z(1))=(0.690, 0.289)\mathbf{a}^{(1)} = \sigma\big(\mathbf{z}^{(1)}\big) = (0.690,\ 0.289).

NoteLesson 20 has a single unit and writes z=w⋅x+bz = \mathbf{w}\cdot\mathbf{x} + b and a=g(z)a = g(z), with no layer number. The pre-activation zz is not a z-score.

Introduced in Lesson 21 (as a(1)\mathbf{a}^{(1)}) and Lesson 22; in general form in Lesson 25.

gg, ReLU⁡\operatorname{ReLU}, tanh⁡\tanh

gg is an activation, applied to each entry on its own. Level 4 uses three: the sigmoid σ\sigma; tanh, the same S shape running from −1 to 1, tanh⁡z=2σ(2z)−1\tanh z = 2\sigma(2z) - 1; and ReLU⁡(z)=max⁡(0,z)\operatorname{ReLU}(z) = \max(0, z), which keeps positive values and turns negative ones into 0.

ExampleReLU⁡(−2)=0, ReLU⁡(3)=3, tanh⁡1≈0.76\operatorname{ReLU}(-2) = 0,\ \operatorname{ReLU}(3) = 3,\ \tanh 1 \approx 0.76

Noteg′g' is the activation’s slope, used in backpropagation. σ\sigma here is the sigmoid, not a standard deviation.

Introduced in Lesson 20.

δ(ℓ)\boldsymbol{\delta}^{(\ell)}

The error signal of layer ℓ\ell: how much the loss changes as each entry of its pre-activation changes, δ(ℓ)=∂L/∂z(ℓ)\boldsymbol{\delta}^{(\ell)} = \partial\mathcal{L}/\partial\mathbf{z}^{(\ell)}. Backpropagation finds it from the output backwards, δ(ℓ)=(W(ℓ+1)⊤δ(ℓ+1))⊙g′(z(ℓ))\boldsymbol{\delta}^{(\ell)} = \big(\mathbf{W}^{(\ell+1)\top}\boldsymbol{\delta}^{(\ell+1)}\big) \odot g'\big(\mathbf{z}^{(\ell)}\big), and each weight’s gradient is the error signal of the unit it feeds times the input it multiplies.

ExampleIn Lesson 22, with a sigmoid output, binary cross-entropy and y=1y = 1: δ(2)=p^−y=0.625−1=−0.375\delta^{(2)} = \hat{p} - y = 0.625 - 1 = -0.375.

NoteA layer with one unit has a plain δ\delta. Lesson 8’s bold δ\boldsymbol{\delta} is a small move of the parameters, and Lesson 28’s δt\boldsymbol{\delta}_t is the error signal at time step tt.

Introduced in Lesson 22; used again in Lesson 25.

β\beta, m\mathbf{m}

Momentum keeps a running sum of past gradients g\mathbf{g} and steps along it: m←β m+g\mathbf{m} \leftarrow \beta\,\mathbf{m} + \mathbf{g}, then w←w−η m\mathbf{w} \leftarrow \mathbf{w} - \eta\,\mathbf{m}. β\beta, between 0 and 1, sets how long old gradients are remembered.

ExampleIf every gradient were the same g\mathbf{g}, with β=0.9\beta = 0.9 the sum would grow towards g/(1−β)=10 g\mathbf{g}/(1 - \beta) = 10\,\mathbf{g}.

NoteLesson 27’s bold m\mathbf{m} holds the pooled values instead.

Introduced in Lesson 24.

β1\beta_1, β2\beta_2, ϵ\epsilon

Adam keeps two running means for each weight: m\mathbf{m} of the gradients, with factor β1\beta_1, and s\mathbf{s} of the squared gradients, with factor β2\beta_2. Both start at 0, so after tt steps it divides them by 1−β1 t1 - \beta_1^{\,t} and 1−β2 t1 - \beta_2^{\,t}, then steps by η m^/(s^+ϵ)\eta\,\hat{\mathbf{m}}/(\sqrt{\hat{\mathbf{s}}} + \epsilon), weight by weight. ϵ\epsilon is a tiny number that only guards against dividing by 0. Every lesson here uses β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999 and ϵ=10−8\epsilon = 10^{-8}.

ExampleAt the first step m=(1−β1) g=0.1 g\mathbf{m} = (1 - \beta_1)\,\mathbf{g} = 0.1\,\mathbf{g}, and dividing by 1−β1=0.11 - \beta_1 = 0.1 gives back g\mathbf{g}. So the first step moves each weight by about η, whatever the size of its gradient.

NoteLesson 24 writes ϵ\epsilon; Lesson 27 and Lesson 28 write ε\varepsilon. Both are this tiny constant, not the noise ε\varepsilon of Lesson 1. Lesson 24’s RMSProp keeps s\mathbf{s} alone, with factor β2\beta_2.

Introduced in Lesson 24; used again in Lesson 23, Lesson 27 and Lesson 28, which train with Adam.

ss, pp, ff

A convolution slides an f×ff \times f filter across an n×nn \times n image. The stride ss is how far the filter moves each time, and the padding pp is the number of rings of zeros added around the image. The feature map then has ⌊n+2p−fs⌋+1\left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1 rows and columns. Convolution here is cross-correlation (no flip): the filter is laid on the image as written.

ExampleA 12 × 12 image and a 3 × 3 filter: ⌊(12+0−3)/1⌋+1=10\lfloor (12 + 0 - 3)/1 \rfloor + 1 = 10; with stride 2, ⌊(12+0−3)/2⌋+1=5\lfloor (12 + 0 - 3)/2 \rfloor + 1 = 5; with padding 1, ⌊(12+2−3)/1⌋+1=12\lfloor (12 + 2 - 3)/1 \rfloor + 1 = 12, the size of the image.

NoteHere ss is not a sample SD, pp is not a probability or a polynomial degree, and ff is not a function.

Introduced in Lesson 26; used again in Lesson 27.

Wk\mathbf{W}_k, bkb_k, boutb_{\text{out}}

In Lesson 27’s network, filter kk has a 3 × 3 grid of weights Wk\mathbf{W}_k and a bias bkb_k. AkA_k is its feature map after ReLU and MkM_k that map after max pooling. The pooled values, in one vector m\mathbf{m}, feed the output unit, which has weights v\mathbf{v} and bias boutb_{\text{out}}: z=v⋅m+boutz = \mathbf{v}\cdot\mathbf{m} + b_{\text{out}}.

ExampleTwo filters of 9 weights, their 2 biases, 18 output weights and the output bias: 2×9+2+18+1=392 \times 9 + 2 + 18 + 1 = 39.

NoteHere kk numbers the filters, and v\mathbf{v} is not Lesson 19’s direction.

Introduced in Lesson 27.

ht\mathbf{h}_t, Wh\mathbf{W}_h

The hidden state ht\mathbf{h}_t: what a recurrent network carries from one time step to the next. It starts at h0=0\mathbf{h}_0 = \mathbf{0} and is updated from the new input and the old state, ht=tanh⁡(wxxt+Whht−1+b)\mathbf{h}_t = \tanh\big(\mathbf{w}_x x_t + \mathbf{W}_h \mathbf{h}_{t-1} + \mathbf{b}\big). Wh\mathbf{W}_h holds the weights from the old state to the new, the same at every step. wx\mathbf{w}_x and b\mathbf{b} are the input weights and biases, and wy\mathbf{w}_y and byb_y turn ht\mathbf{h}_t into the output’s score.

ExampleOne unit with wx=1\mathbf{w}_x = 1, Wh=0.5\mathbf{W}_h = 0.5 and b=0\mathbf{b} = 0, fed x1=1x_1 = 1 then x2=0x_2 = 0: h1=tanh⁡1≈0.76\mathbf{h}_1 = \tanh 1 \approx 0.76, then h2=tanh⁡(0.5×0.76)≈0.36\mathbf{h}_2 = \tanh(0.5 \times 0.76) \approx 0.36. The 1 is still felt a step later.

NoteBold ht\mathbf{h}_t is a vector of hidden values, not Lesson 5’s small step hh.

Introduced in Lesson 28.

E\mathbf{E}, dd, P\mathbf{P}

The embedding matrix E\mathbf{E} has one row per token, each of width dd: row ii, written Ei\mathbf{E}_i, is the vector learned for token ii. Multiplying the one-hot vector ei\mathbf{e}_i by E\mathbf{E} picks out that row, so an embedding is a table look-up. The position embedding P\mathbf{P} has one row per position, and a transformer starts position tt as xt=Eit+Pt\mathbf{x}_t = \mathbf{E}_{i_t} + \mathbf{P}_t, where iti_t is the token there.

ExampleIn Lesson 30, 27 words with d=2d = 2 give E\mathbf{E} 27×2=5427 \times 2 = 54 numbers, so each word is a point in the plane. Lesson 33 and Lesson 34 use d=32d = 32.

NoteLesson 32 writes xt=Ext+Pt\mathbf{x}_t = \mathbf{E}_{x_t} + \mathbf{P}_t, with the token xtx_t as the row number. Lesson 30 writes word cc’s row as wc\mathbf{w}_c, and its output matrix Wout\mathbf{W}_{\text{out}} has rows uj\mathbf{u}_j. Bold E\mathbf{E} is not the expected value E\mathbb{E} or the number of epochs EE, and bold P\mathbf{P} is not a probability P(⋅)P(\cdot).

E\mathbf{E} introduced in Lesson 30, P\mathbf{P} in Lesson 32; both used again in Lesson 33 and Lesson 34.

softmax⁡\operatorname{softmax}, τ\tau

Softmax turns a list of scores into probabilities: softmax⁡(z)i=exp⁡(zi)∑jexp⁡(zj)\operatorname{softmax}(\mathbf{z})_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}. Every output is positive, they add up to 1, and a larger score gets a larger share. The temperature τ\tau, said “tau”, divides the scores first, softmax⁡(z/τ)\operatorname{softmax}(\mathbf{z}/\tau): τ\tau below 1 sharpens the probabilities towards the largest score, τ\tau above 1 flattens them towards equal shares, and τ=1\tau = 1 changes nothing.

Examplesoftmax⁡(0,ln⁡3)=(0.25, 0.75)\operatorname{softmax}(0, \ln 3) = (0.25,\ 0.75), since eln⁡3=3e^{\ln 3} = 3. At τ=2\tau = 2 it is flatter: softmax⁡((0,ln⁡3)/2)≈(0.37, 0.63)\operatorname{softmax}\big((0, \ln 3)/2\big) \approx (0.37,\ 0.63).

NoteWith two scores, softmax⁡(0,z)\operatorname{softmax}(0, z) gives σ(z)\sigma(z) for the second, so softmax is the many-class form of the sigmoid of Lesson 12. Lesson 31 reshapes probabilities rather than scores, qτ(v)∝p(v)1/τq_\tau(v) \propto p(v)^{1/\tau}, which is the same as the softmax of ln⁡p(v)/τ\ln p(v)/\tau. The letter is τ\tau because TT is already a length.

Introduced in Lesson 30; temperature in Lesson 31. Used again in Lesson 32, Lesson 33 and Lesson 34.

Q\mathbf{Q}, K\mathbf{K}, V\mathbf{V}, A\mathbf{A}, dkd_k

Attention. Each position’s vector is mapped to a query q\mathbf{q}, a key k\mathbf{k} and a value v\mathbf{v}, each with dkd_k entries. Stacked one position per row, they form the matrices Q\mathbf{Q}, K\mathbf{K} and V\mathbf{V}. A query scores every key by q⋅k/dk\mathbf{q}\cdot\mathbf{k}/\sqrt{d_k}, softmax along each row turns the scores into the attention weights A\mathbf{A}, and each output is the weighted average of the values: Attention⁡(Q,K,V)=softmax⁡ ⁣(QK⊤dk)V=AV\operatorname{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \operatorname{softmax}\!\Big(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_k}}\Big)\mathbf{V} = \mathbf{A}\mathbf{V}. With a causal mask, position tt may attend only to positions s≤ts \le t.

ExampleTwo of the keys and values of Lesson 32, k1=(2,0)\mathbf{k}_1 = (2, 0) and k2=(0,2)\mathbf{k}_2 = (0, 2), with v1=(2,2)\mathbf{v}_1 = (2, 2) and v2=(−2,2)\mathbf{v}_2 = (-2, 2), and the query q=(1,0)\mathbf{q} = (1, 0), so dk=2d_k = 2: s1=q⋅k1/2≈1.41, s2=0s_1 = \mathbf{q}\cdot\mathbf{k}_1/\sqrt{2} \approx 1.41,\ s_2 = 0, then (a1,a2)=softmax⁡(1.41,0)≈(0.8, 0.2)(a_1, a_2) = \operatorname{softmax}(1.41, 0) \approx (0.8,\ 0.2) and o=a1v1+a2v2≈(1.22, 2)\mathbf{o} = a_1 \mathbf{v}_1 + a_2 \mathbf{v}_2 \approx (1.22,\ 2).

NoteBold V\mathbf{V} holds values; the vocabulary is the curly V\mathcal{V}. Lesson 32 writes the maps as WQ\mathbf{W}_Q, WK\mathbf{W}_K and WV\mathbf{W}_V, with one position as a column, qt=WQxt\mathbf{q}_t = \mathbf{W}_Q \mathbf{x}_t; Lesson 33 writes WhQ\mathbf{W}^{Q}_h for head hh and multiplies rows on the right. Bold K\mathbf{K} is not the number of clusters KK.

Introduced in Lesson 32; used again in Lesson 33 and Lesson 34.

HH, hh, WO\mathbf{W}^{O}

Multi-head attention runs HH attention heads side by side, numbered h=1,…,Hh = 1, \dots, H. Each head has its own query, key and value maps and its own weights Ah\mathbf{A}_h. Their outputs are placed side by side and mapped back to width dd by the matrix WO\mathbf{W}^{O}.

ExampleLesson 33 has d=32d = 32 and two heads, each of width dk=d/H=32/2=16d_k = d/H = 32/2 = 16.

NoteHere HH counts heads. In Lesson 31 and Lesson 34, HH is the cross-entropy in bits, and in Lesson 8 it is the Hessian.

Introduced in Lesson 33; Lesson 34 also uses two heads.

LN⁡\operatorname{LN}, γ\boldsymbol{\gamma}, β\boldsymbol{\beta}

Layer normalisation rescales one position’s dd numbers to mean 0 and variance 1, then applies a learned scale γ\boldsymbol{\gamma} and shift β\boldsymbol{\beta}: LN⁡(x)=γ⊙x−μs2+ε+β\operatorname{LN}(\mathbf{x}) = \boldsymbol{\gamma} \odot \frac{\mathbf{x} - \mu}{\sqrt{s^2 + \varepsilon}} + \boldsymbol{\beta}. Here μ\mu and s2s^2 are the mean and the population variance (divide by dd, not d−1d - 1) of the dd entries of x\mathbf{x}, and ⊙\odot multiplies entry by entry. The tiny constant ε\varepsilon only guards against dividing by 0.

ExampleFor x=(1,2,3)\mathbf{x} = (1, 2, 3), with γ=1\boldsymbol{\gamma} = \mathbf{1}, β=0\boldsymbol{\beta} = \mathbf{0} and ε\varepsilon left out: μ=2\mu = 2, s2=2/3s^2 = 2/3 and LN⁡(x)≈(−1.22, 0, 1.22)\operatorname{LN}(\mathbf{x}) \approx (-1.22,\ 0,\ 1.22).

Noteβ\boldsymbol{\beta} and γ\boldsymbol{\gamma} here are vectors learned in training: not momentum β\beta, Adam’s β1\beta_1 and β2\beta_2, or the eigenvalues γ\gamma. Lesson 33 uses ε=10−5\varepsilon = 10^{-5}, a different constant from Adam’s, and writes LN⁡1\operatorname{LN}_1, LN⁡2\operatorname{LN}_2 and LN⁡f\operatorname{LN}_f for its three layer norms. Each position is normalised on its own, not across a batch.

Introduced in Lesson 33; used again in Lesson 34.

πk\pi_k, μk\boldsymbol{\mu}_k, Σk\boldsymbol{\Sigma}_k, rikr_{ik}

A generative model learns the data distribution p(x)p(\mathbf{x}). A mixture of KK Gaussians writes it as p(x)=∑k=1Kπk N(x∣μk,Σk)p(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k). Component kk has a weight πk\pi_k, a mean μk\boldsymbol{\mu}_k and a covariance matrix Σk\boldsymbol{\Sigma}_k. The weights are at least 0 and add up to 1: πk\pi_k is the share of points the component accounts for (a weight, not 3.14…). In EM, the responsibility rikr_{ik} is the probability that point ii came from component kk, given the current fit; for each point they add up to 1.

ExampleOne dimension, two components with π1=π2=0.5\pi_1 = \pi_2 = 0.5, means 0 and 2 and variance 1. For a point at 0.5, ri1=0.5 N(0.5∣0,1)0.5 N(0.5∣0,1)+0.5 N(0.5∣2,1)r_{i1} = \frac{0.5\,\mathcal{N}(0.5 \mid 0, 1)}{0.5\,\mathcal{N}(0.5 \mid 0, 1) + 0.5\,\mathcal{N}(0.5 \mid 2, 1)}. The two densities differ by a factor e1e^{1}, so ri1=σ(1)≈0.73r_{i1} = \sigma(1) \approx 0.73, with σ\sigma the sigmoid: the point more likely came from the component at 0.

NoteLesson 35 writes Nk=∑irikN_k = \sum_i r_{ik} for the number of points component kk accounts for, and fits πk=Nk/n\pi_k = N_k/n. Bold Σk\boldsymbol{\Sigma}_k is a covariance matrix, not the summation sign; bold μk\boldsymbol{\mu}_k is a component’s mean, as it is a cluster centre in Lesson 18. The constant π\pi still appears inside the normal density.

Introduced in Lesson 35; the mixture is compared again in Lesson 36.

xt\mathbf{x}_t, βt\beta_t, αt\alpha_t, αˉt\bar\alpha_t

In diffusion, x0\mathbf{x}_0 is a data point and xt\mathbf{x}_t the same point after tt noising steps, t=1,…,Tt = 1, \dots, T. The noise schedule βt\beta_t sets how much noise step tt adds. With αt=1−βt\alpha_t = 1 - \beta_t, the running product αˉt=α1α2⋯αt\bar\alpha_t = \alpha_1 \alpha_2 \cdots \alpha_t is said “alpha-bar”. After tt steps, xt=αˉt x0+1−αˉt ε\mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1 - \bar\alpha_t}\,\boldsymbol{\varepsilon}.

ExampleIf β1=β2=0.1\beta_1 = \beta_2 = 0.1, then α1=α2=0.9\alpha_1 = \alpha_2 = 0.9 and αˉ2=0.9×0.9=0.81\bar\alpha_2 = 0.9 \times 0.9 = 0.81, so x2\mathbf{x}_2 keeps αˉ2=0.9\sqrt{\bar\alpha_2} = 0.9 of x0\mathbf{x}_0.

NoteLesson 36 uses T=50T = 50 steps, with βt\beta_t rising evenly from 0.001 to 0.2. There TT is a number of steps, not a transpose; βt\beta_t is not momentum or a layer norm shift, and αt\alpha_t is not Lesson 14’s αi\alpha_i. Lesson 36 writes q(xt∣xt−1)q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) for the noising recipe, not Lesson 31’s qτq_\tau.

Introduced in Lesson 36.

ε\boldsymbol{\varepsilon}, ε^θ\hat{\boldsymbol{\varepsilon}}_\theta

ε∼N(0,I)\boldsymbol{\varepsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) is the noise drawn to make xt\mathbf{x}_t from x0\mathbf{x}_0. The network’s guess of it is ε^θ(xt,t)\hat{\boldsymbol{\varepsilon}}_\theta(\mathbf{x}_t, t), with θ\theta its weights, and training makes E ∥ε−ε^θ(xt,t)∥2\mathbb{E}\,\lVert \boldsymbol{\varepsilon} - \hat{\boldsymbol{\varepsilon}}_\theta(\mathbf{x}_t, t) \rVert^2 small. To generate, it starts from xT∼N(0,I)\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) and steps back to x0\mathbf{x}_0, each step taking out some of the guessed noise and, until the last, adding fresh noise σtz\sigma_t \mathbf{z}.

ExampleIn two dimensions, always guessing ε^=0\hat{\boldsymbol{\varepsilon}} = \mathbf{0} costs E ∥ε∥2=1+1=2\mathbb{E}\,\lVert \boldsymbol{\varepsilon} \rVert^2 = 1 + 1 = 2 on average: each standard normal number has mean square 1.

NoteBold ε\boldsymbol{\varepsilon} is noise, as ε\varepsilon is in Lesson 1; it is not the tiny constant of Adam or of layer normalisation. σt\sigma_t is the standard deviation of the fresh noise (σt2=βt\sigma_t^2 = \beta_t in Lesson 36), not the sigmoid, and the bold z\mathbf{z} it multiplies is a fresh normal draw, not a pre-activation or a z-score. A plain θ\theta here holds the network’s weights, as bold θ\boldsymbol{\theta} does elsewhere.

Introduced in Lesson 36.

Losses and measures of fit

Ways to sum up all the misses in one number. Examples use the least-squares line through (1, 1), (2, 3) and (3, 2), whose residuals are −0.5,1,−0.5-0.5, 1, -0.5.

MSE\mathrm{MSE}

Mean squared error: the average of the squared residuals, 1n∑iei2\frac{1}{n}\sum_{i} e_i^2.

ExampleMSE=(0.25+1+0.25)/3=0.5\mathrm{MSE} = (0.25 + 1 + 0.25)/3 = 0.5

NoteLesson 9 and Lesson 10 write MSEtrain\mathrm{MSE}_{\text{train}} and MSEtest\mathrm{MSE}_{\text{test}} for the MSE on the training and test points.

Introduced in Lesson 4. It appears earlier in the maths of Lesson 1.

MAE\mathrm{MAE}

Mean absolute error: the average size of the residuals, ignoring their sign.

ExampleMAE=(0.5+1+0.5)/3≈0.67\mathrm{MAE} = (0.5 + 1 + 0.5)/3 \approx 0.67

Introduced in Lesson 4.

RMSE\mathrm{RMSE}

Root mean squared error: the square root of the MSE, back in the units of yy.

ExampleRMSE=0.5≈0.71\mathrm{RMSE} = \sqrt{0.5} \approx 0.71

Introduced in Lesson 4.

SSE\mathrm{SSE}

Sum of squared errors: the squared residuals added up, without dividing. MSE=SSE/n\mathrm{MSE} = \mathrm{SSE}/n.

ExampleSSE=0.25+1+0.25=1.5\mathrm{SSE} = 0.25 + 1 + 0.25 = 1.5

Introduced in Lesson 6.

SST\mathrm{SST}

Total sum of squares: the squared distances of each yiy_i from yˉ\bar{y}, added up. It is the SSE of the flat line y^=yˉ\hat{y} = \bar{y}.

ExampleSST=1+1+0=2\mathrm{SST} = 1 + 1 + 0 = 2

Introduced in Lesson 6.

R2R^2

The share of the variation in yy that the fitted line accounts for, 1−SSE/SST1 - \mathrm{SSE}/\mathrm{SST}. For a least-squares fit with an intercept it lies between 0 and 1.

ExampleR2=1−1.5/2=0.25R^2 = 1 - 1.5/2 = 0.25

Introduced in Lesson 6.

rr

The correlation of xx and yy: a number between −1 and 1 that measures how closely the points follow a straight line. For the least-squares line with one input, R2=r2R^2 = r^2.

Exampler=1/2×2=0.5r = 1/\sqrt{2 \times 2} = 0.5, and 0.52=0.250.5^2 = 0.25.

NoteLesson 19 writes rr for the correlation of two features and RR for their correlation matrix. Lesson 16’s ρ\rho (rho) is the correlation between two trees’ predictions. Lesson 17’s rmir_{mi} is a residual, not a correlation.

Introduced in Lesson 6.

RR, TT, ∣T∣|T|

The training set RR and the test set TT. Bars around a set mean the number of points in it.

Example∣R∣=28, ∣T∣=12|R| = 28,\ |T| = 12

Introduced in Lesson 9.

CVk\mathrm{CV}_k

The kk-fold cross-validation score: the average of the kk validation MSEs, one for each part held out.

ExampleCVk=1k∑j=1kMSEj\mathrm{CV}_k = \frac{1}{k}\sum_{j=1}^{k} \mathrm{MSE}_j, with k=5k = 5 in Lesson 9.

Introduced in Lesson 9.

MM

The number of training sets drawn to estimate bias and variance.

ExampleM=200M = 200

NoteIn Lesson 17, MM is the number of boosting rounds instead, from 0 to 300 on its slider.

Introduced in Lesson 10.

accuracy\text{accuracy}

The share of cases a classifier labels correctly: accuracy=1n∑i=1n1[y^i=yi]\text{accuracy} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}[\hat{y}_i = y_i].

Example45 right out of 50: accuracy=45/50=0.9\text{accuracy} = 45/50 = 0.9.

Introduced in Lesson 11.

L(b,w)L(b, w)

The mean log-loss of logistic regression: L(b,w)=−1n∑i=1n[ yiln⁡p^i+(1−yi)ln⁡(1−p^i)]L(b, w) = -\frac{1}{n}\sum_{i=1}^{n} \Big[\, y_i \ln \hat{p}_i + (1 - y_i)\ln(1 - \hat{p}_i) \Big]. Each case is charged minus the log of the probability the model gave to what happened, so a confident mistake costs a lot.

ExampleA pass (y=1y = 1) given p^=0.88\hat{p} = 0.88 costs −ln⁡0.88≈0.13-\ln 0.88 \approx 0.13.

Introduced in Lesson 12.

precision\text{precision}, recall\text{recall}

With TP, FP and FN the counts of true positives, false positives and false negatives, precision, TP/(TP+FP)TP/(TP + FP), is the share of predicted 1s that are right, and recall, TP/(TP+FN)TP/(TP + FN), is the share of real 1s that are found.

ExampleWith 8 true positives, 2 false positives and 4 false negatives: precision=8/(8+2)=0.8\text{precision} = 8/(8 + 2) = 0.8 and recall=8/(8+4)≈0.67\text{recall} = 8/(8 + 4) \approx 0.67.

Introduced in Lesson 12.

G=1−∑kpk2G = 1 - \sum_k p_k^2

The Gini impurity of a group of cases, with pkp_k the share of class kk in the group. It is 0 for a pure group; with two classes it is 2p(1−p)2p(1 - p), at most 0.5. A split is scored by GsplitG_{\text{split}}, the Gini of its two sides weighted by their sizes.

ExampleThree circles and one triangle: G=1−(3/4)2−(1/4)2=0.375G = 1 - (3/4)^2 - (1/4)^2 = 0.375.

Introduced in Lesson 15.

inertia\text{inertia}

The k-means loss: the sum of the squared distances from each point to its own centre, ∑i=1n∥xi−μc(i)∥2\sum_{i=1}^{n} \lVert \mathbf{x}_i - \boldsymbol{\mu}_{c(i)} \rVert^2.

ExampleThe three points with one centre at their mean (2,2)(2, 2): inertia=2+1+1=4\text{inertia} = 2 + 1 + 1 = 4.

Introduced in Lesson 18.

eke_k

The share of the total variance explained by principal component kk: ek=γk/(γ1+γ2)e_k = \gamma_k/(\gamma_1 + \gamma_2).

Examplee1=1.5/(1.5+0.5)=0.75e_1 = 1.5/(1.5 + 0.5) = 0.75 for the three points.

NoteIt is not a residual eie_i.

Introduced in Lesson 19.

L\mathcal{L}

The loss in Level 4, a curly L. Each page says whether it is the loss of one example or the mean over a batch or the training set. Binary cross-entropy is the log-loss of Lesson 12: for one example, L=−[ yln⁡p^+(1−y)ln⁡(1−p^)]\mathcal{L} = -\big[\, y \ln \hat{p} + (1 - y)\ln(1 - \hat{p}) \big].

ExampleA class 0 case given p^=σ(2)≈0.88\hat{p} = \sigma(2) \approx 0.88 costs L=−ln⁡(1−0.88)≈2.13\mathcal{L} = -\ln(1 - 0.88) \approx 2.13: a confident mistake.

NoteLesson 25’s LB\mathcal{L}_B is the mean over a batch; Lesson 28’s LT\mathcal{L}_T is the loss at the last time step alone. In Level 5, L\mathcal{L} is a mean cross-entropy, minus the log of the probability given to the right answer: over word pairs in Lesson 30, steps in Lesson 32, positions in Lesson 33 and windows of text in Lesson 34.

Introduced in Lesson 20.

HH, log⁡2\log_2

Cross-entropy in bits per character: the mean of −log⁡2p-\log_2 p over the characters of a text, where pp is the probability the model gave the character that really came next. log⁡2\log_2 is the log to base 2, so a sure, right guess costs 0 bits and each halving of pp costs one more bit. Lower is better.

ExampleA uniform guess over the 28 symbols gives p=1/28p = 1/28 every time and costs log⁡228≈4.81\log_2 28 \approx 4.81 bits per character.

NoteIt is the many-class form of Lesson 12’s log-loss, in base 2 instead of natural logs: −log⁡2p=−ln⁡p/ln⁡2-\log_2 p = -\ln p/\ln 2. Lesson 34’s training loss L\mathcal{L} uses natural logs. Here HH is not the Hessian of Lesson 8 or the number of heads of Lesson 33.

Introduced in Lesson 31; used again in Lesson 34.

L(θ)L(\theta), ℓ(θ)\ell(\theta)

The likelihood of parameters θ\theta is the density of the data under them, the values taken as independent: L(θ)=∏i=1np(xi∣θ)L(\theta) = \prod_{i=1}^{n} p(x_i \mid \theta). Its log, the log-likelihood ℓ(θ)=∑iln⁡p(xi∣θ)\ell(\theta) = \sum_i \ln p(x_i \mid \theta), has the same maximum and is easier to work with. Fitting by maximum likelihood makes them as large as possible.

ExampleTwo points with densities 0.5 and 0.2: L(θ)=0.5×0.2=0.1L(\theta) = 0.5 \times 0.2 = 0.1 and ℓ(θ)=ln⁡0.1≈−2.3\ell(\theta) = \ln 0.1 \approx -2.3.

NoteHere LL is a likelihood, made large, not a loss LL, made small, and ℓ\ell is not a layer number or a leaf. Lesson 35’s scores are ℓ/n\ell/n, the mean log-likelihood per point. A density can be above 1, so ℓ\ell can have either sign.

Introduced in Lesson 35.

Calculus and other operations

∑\sum

Sum, a capital sigma: add up the terms as ii runs over its range.

Example∑i=13xi=1+2+3=6\sum_{i=1}^{3} x_i = 1 + 2 + 3 = 6

Introduced in Lesson 1’s maths.

∏\prod, ∝\propto

∏\prod, a capital pi, multiplies the terms, as ∑\sum adds them. ∝\propto means “proportional to”: equal up to a factor that is the same for every class.

Example∏j=13j=1×2×3=6\prod_{j=1}^{3} j = 1 \times 2 \times 3 = 6. In Lesson 13, P(c∣x)∝P(c)∏j=1dp(xj∣c)P(c \mid \mathbf{x}) \propto P(c)\prod_{j=1}^{d} p(x_j \mid c).

Introduced in Lesson 13.

1[⋅]\mathbf{1}[\cdot]

The indicator: 1 when the statement inside is true and 0 when it is false. Adding it up counts the cases where the statement holds.

Example1[3>2]=1, 1[2>3]=0\mathbf{1}[3 > 2] = 1,\ \mathbf{1}[2 > 3] = 0

Introduced in Lesson 11.

∈\in, ∉\notin

“Is one of” and “is not one of” a set.

Exampleyi∈{0,1}y_i \in \{0, 1\}; i∉Sbi \notin S_b says row ii is not in tree bb’s sample.

Introduced in Lesson 11; ∉\notin in Lesson 16.

∣e∣|e|

Absolute value: the size of a number, without its sign.

Example∣−0.5∣=0.5|-0.5| = 0.5

Introduced in Lesson 4.

∥x∥\lVert \mathbf{x} \rVert

The norm, or length, of a vector: the square root of the sum of its squared entries. The distance between two points is the norm of their difference.

Example∥(3,4)∥=32+42=5\lVert (3, 4) \rVert = \sqrt{3^2 + 4^2} = 5

NoteFor a matrix, Lesson 23’s ∥W∥2\lVert \mathbf{W} \rVert^2 adds up the squares of all its entries. Lesson 25’s ∥W∥2\lVert \mathbf{W} \rVert_2 is the most W\mathbf{W} can stretch the length of a vector.

Introduced in Lesson 2; used again in Lesson 8.

f′(x)f'(x)

The derivative of ff at xx, said “f-prime” and also written df/dxdf/dx: the slope of the curve there, f′(x)=lim⁡h→0f(x+h)−f(x)hf'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}. The limit is the value the fraction gets as close to as you like as hh shrinks. Positive means uphill to the right. f′′f'' is the derivative of the derivative.

Examplef(x)=x2f(x) = x^2 has f′(x)=2xf'(x) = 2x, so f′(3)=6f'(3) = 6.

NoteLesson 7 writes L′(w)L'(w) for the slope of the loss.

Introduced in Lesson 5.

ln⁡\ln, eze^{z}

The natural log and its reverse, the exponential, with e≈2.718e \approx 2.718. ln⁡ez=z\ln e^{z} = z, and a log turns a product into a sum. Lesson 13 writes log⁡\log and exp⁡\exp; on this site these are the same natural log and exponential.

Exampleln⁡2≈0.693\ln 2 \approx 0.693, and e0=1e^{0} = 1.

Introduced in Lesson 12.

∂g/∂u\partial g/\partial u

A partial derivative: the slope as one input changes while the others are held fixed.

Exampleg(u,v)=u2+3v2g(u, v) = u^2 + 3v^2 has ∂g/∂u=2u\partial g/\partial u = 2u.

Introduced in Lesson 5; used in Lesson 6 and Lesson 8.

∇g\nabla g

The gradient, said “grad”: all the partial derivatives in one vector. Where it is not zero, it points in the direction in which gg rises fastest, so gradient descent steps the other way.

Example∇g=(2u, 6v)\nabla g = (2u,\ 6v), which is (2,6)(2, 6) at (1,1)(1, 1).

Introduced in Lesson 5; used again in Lesson 8.

a⋅b\mathbf{a} \cdot \mathbf{b}, δ\boldsymbol{\delta}

The dot product: multiply matching entries and add. In Lesson 8, δ\boldsymbol{\delta} (delta) is a small move of the parameters, and the loss changes by about ∇L⋅δ\nabla L \cdot \boldsymbol{\delta}.

Example(1,2)⋅(3,4)=1×3+2×4=11(1, 2) \cdot (3, 4) = 1 \times 3 + 2 \times 4 = 11

NoteLevel 4’s δ(ℓ)\boldsymbol{\delta}^{(\ell)} is something else, a layer’s error signal.

Used in Lesson 8, and from Lesson 12 on, in scores such as w⋅x+b\mathbf{w}\cdot\mathbf{x} + b.

arg⁡min⁡\arg\min

The input that makes something smallest, not the smallest value itself. That value is written min⁡\min.

Examplearg⁡min⁡c (c−2)2=2\arg\min_c\,(c - 2)^2 = 2, while min⁡c (c−2)2=0\min_c\,(c - 2)^2 = 0.

Notearg⁡max⁡\arg\max is the input that makes something largest. Lesson 11 picks k^\hat{k} that way.

Introduced in Lesson 1’s maths; used again in Lesson 9, Lesson 11 and Lesson 18.

X⊤X^{\top}

The transpose: the matrix with its rows turned into columns.

Example(12)⊤=(12)\begin{pmatrix} 1 & 2 \end{pmatrix}^{\top} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}

NoteLesson 21, Lesson 22 and Lesson 28 print the raised T upright; it is the same transpose. An italic TT on the line is a number: the test set of Lesson 9 or the number of time steps in Lesson 28.

Used in the maths of Lesson 8, Lesson 10 and Lesson 19, and in Level 4 from Lesson 21 on.

tr⁡\operatorname{tr}

The trace of a square matrix: the sum of the numbers on its diagonal. It equals the sum of the eigenvalues.

Exampletr⁡S=1+1=2=γ1+γ2\operatorname{tr} S = 1 + 1 = 2 = \gamma_1 + \gamma_2 for the three points.

Introduced in Lesson 19.

⊙\odot

The entry-by-entry product of two vectors of the same length.

Example(1,2)⊙(3,4)=(3,8)(1, 2) \odot (3, 4) = (3, 8)

Introduced in Lesson 22; used again in Lesson 25 and Lesson 28.

X⋆WX \star \mathbf{W}

Cross-correlation, which networks call convolution: lay the filter W\mathbf{W} on a patch of the image XX, multiply each pixel by the weight on top of it and add up, then slide to the next position: (X⋆W)[r,c]=∑i,jX[r+i, c+j] W[i,j](X \star \mathbf{W})[r, c] = \sum_{i,j} X[r + i,\, c + j]\,\mathbf{W}[i, j].

ExampleLesson 26’s edge filter has columns of −1-1, 00 and 11. On a patch whose right column is three 1s and the rest 0s, (X⋆W)[0,0]=1+1+1=3(X \star \mathbf{W})[0, 0] = 1 + 1 + 1 = 3.

NoteThe convolution of signal processing flips the filter first. Here it is not flipped.

Introduced in Lesson 26.

⌊⋅⌋\lfloor \cdot \rfloor

The floor: round down to a whole number.

Example⌊4.5⌋=4\lfloor 4.5 \rfloor = 4

Introduced in Lesson 26.

cos⁡θ\cos\theta

Cosine similarity: for two vectors a\mathbf{a} and b\mathbf{b} at angle θ\theta, cos⁡θ=a⋅b∥a∥ ∥b∥\cos\theta = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert\, \lVert \mathbf{b} \rVert}. It is 1 when they point the same way, 0 at right angles and −1 when they point opposite ways; their lengths do not matter.

ExampleFor (1,0)(1, 0) and (1,1)(1, 1): cos⁡θ=11×2≈0.71\cos\theta = \frac{1}{1 \times \sqrt{2}} \approx 0.71, an angle of 45°.

NoteHere θ\theta is an angle, as in Lesson 19, not a model’s parameters.

Introduced in Lesson 30.

Letters used only in passing

Some letters stand for something only inside one explanation, and are defined where they appear: cc for a constant being tried out (Lesson 3 and Lesson 4) and for a level of the loss (Lesson 8); c1c_1 and c2c_2 for how far the start is from the bottom along the bowl’s steep and gentle directions (Lesson 8); dd for how far one value moves (Lesson 3); hh for a small step along xx, and nn as the power in xnx^n (Lesson 5); h1h_1 and h2h_2 for the heights of the two handles (Lesson 6); h(u)h(u) for a curve with two dips (Lesson 7); pp, qq, kk and rr in the arguments about MAE and squaring (Lesson 4); qq for the factor the distance to the bottom shrinks by at each step, with dkd_k (bold in Lesson 8) for that distance (Lesson 7 and Lesson 8); θk\boldsymbol{\theta}_k for the parameters after kk steps (Lesson 8); x0x_0 for one chosen input (Lesson 10); cc for the factor a feature is multiplied by (Lesson 11); tt for a probability threshold, and TP, FP and FN for counts of right and wrong predictions (Lesson 12); ncn_c, nBn_B and nc,Bn_{c,B} for counts of cases, scs_c for a class’s log score and mm for the largest score (Lesson 13); φ\varphi for the tilt of a line, bold n\mathbf{n} for the arrow at right angles to it, cc for its shift and d0d_0, d1d_1 for the distances to the nearest point of each class (Lesson 14); jj and tt for the feature and threshold of a split, nLn_L and nRn_R for the cases sent each way, and ℓ\ell for a leaf holding nℓn_\ell cases (Lesson 15); ρ\rho for the correlation between trees’ predictions (Lesson 16); ss for a stump’s split point (Lesson 17); CkC_k and nkn_k for the points in cluster kk and how many there are (Lesson 18); and did_i for a point’s distance from a line, x^i\hat{\mathbf{x}}_i for the point rebuilt from one number, and RR for the correlation matrix (Lesson 19).

In Level 4: cc for a level of a unit’s score (Lesson 20); jj for a hidden unit and wj\mathbf{w}_j for the weights into it (Lesson 21); hh for a small nudge to one weight, and aina_{\text{in}} for the input a weight multiplies (Lesson 22); ee for an epoch, e^\hat{e} for the epoch with the lowest validation loss, gB\mathbf{g}_B for the gradient on a batch and s2s^2 for the variance of the per-point gradients (Lesson 23); uu and vv for positions along and across the valley, tt for Adam’s step count, and GG for the grid of learning rates tried, with η⋆\eta^\star the best (Lesson 24); ninn_{\text{in}} and noutn_{\text{out}} for the numbers of inputs and units of a layer, vv for the variance of the starting weights, D(k)\mathbf{D}^{(k)} for the diagonal matrix of slopes g′g', and ff for the block a residual connection skips over (Lesson 25); rr and cc for a row and a column, and dd for a shift (Lesson 26); AkA_k and MkM_k for a filter’s map before and after pooling (Lesson 27); and kk for how many steps back the network must look, at\mathbf{a}_t for the recurrent layer’s pre-activation, Js\mathbf{J}_s for the change in one hidden state per change in the one before, and ct\mathbf{c}_t, ft\mathbf{f}_t, it\mathbf{i}_t and ot\mathbf{o}_t for an LSTM’s cell state and gates (Lesson 28).

Several of these clash with letters above:

Letters with more than one job

From Level 3 on, a few letters take on a second job. The lesson always says which one it means.