ML Explorer · Reference
Notation
Every symbol the lessons use, what it means, and the lesson that introduces it.
The lessons keep one letter for one idea wherever they can. Where a lesson uses a letter differently, the entry says so in a note.
Most examples use the same three points, = (1, 1), (2, 3) and (3, 2). Their least-squares line is .
Data and examples
What the lessons learn from: inputs, answers and the misses in between.
-
An input value, also called a feature: the thing the model is given.
ExampleIn Lesson 1, is the hours a student studied.
Introduced in Lesson 1.
-
The answer that goes with an input, also called the target or label: the thing the model should predict. It is never one of the features.
ExampleIn Lesson 1, is the exam mark.
Introduced in Lesson 1.
-
A class label. In Level 3 the answer is often a class rather than an amount, and the two classes are called 0 and 1. These are names: class 1 is not “more” than class 0. On the plots, class 0 is a cyan circle and class 1 a violet triangle.
Example says that case belongs to class 0 or class 1.
NoteLesson 14 writes the classes as (that is, ), so that one inequality covers both sides of the line. Lesson 18’s data have no labels at all.
Introduced in Lesson 11.
-
A prediction of , said “y-hat”. A hat marks a value worked out from data rather than observed.
Example
NoteFor a classifier, is a predicted class, 0 or 1 (Lesson 11 on). Lesson 19’s is a point rebuilt from a single number.
Introduced in Lesson 1.
- ,
-
is the number of examples. numbers them, from 1 to .
ExampleFor the three points, .
Introduced in Lesson 1.
- ,
-
The input and the answer of example .
Example
Introduced in Lesson 1.
-
The residual of example : how far the answer is from the prediction. It is positive when the point lies above the line.
Example
NoteLesson 17 writes for the residual after rounds of boosting. For class labels a difference has no useful meaning (Lesson 11).
-
Noise: everything the rule does not capture. In the lessons’ made-up data it is the random part added to each .
Example
NoteLesson 4 and Lesson 9 write for the noise on example .
Introduced in Lesson 1.
-
The number of features each example has.
ExampleIn Lesson 2, examples have features each.
NoteThe maths of Lesson 8 writes for the number of features. In Lesson 10, is the degree of a polynomial instead. Lesson 13 writes for the number of features, as here, but in Lesson 15 is the depth of a tree.
Introduced in Lesson 2.
-
A vector: a list of numbers written in bold. It is also a point you can draw.
Example
NoteLesson 2 also names two points and . Those bold letters are points, not the intercept and slope and .
Introduced in Lesson 2.
-
All the features of example , as one vector.
Example
Introduced in Lesson 2.
-
The value of feature for example . With two subscripts, the first is the example and the second the feature.
Example is example 2’s value of feature 1.
Introduced in Lesson 2.
-
Inside one vector, a single subscript numbers its entries: the features of one example.
Example. In Lesson 8’s maths, .
NoteSo can mean example 2’s input (as in ) or feature 2 of one vector; the text says which. In Lesson 5, and are two points on a line.
Introduced in Lesson 2.
-
The set of all pairs of real numbers: every point of a flat plane. More generally, holds every list of real numbers.
Example: each example is a point in the plane.
Introduced in Lesson 2.
- ,
-
The data matrix: one row per example and one column per feature, so rows and columns.
NoteLesson 2 writes it in bold, . The maths of Lesson 8 and Lesson 10 writes and puts a column of 1s first, for the intercept. In Lesson 10 the other columns are powers of .
Introduced in Lesson 2.
-
All the answers in one vector, .
Example for the three points.
- , ,
-
A batch is the small set of training points that one training step uses, and is how many points it holds. One pass through all training points is an epoch, and is the number of epochs. Each epoch the points are shuffled and cut into batches.
ExampleWith points in batches of , an epoch is steps.
NoteLesson 16’s is the number of trees, and Lesson 13’s counts the cases in a band. Pages that need no symbol give the batch size in words.
Introduced in Lesson 23.
-
An image as a grid of numbers: is the pixel in row and column , both counted from 0 at the top left.
ExampleLesson 26’s images are 12 × 12, so . Lesson 27’s are 8 × 8.
NoteAn image is not the data matrix , which holds one example per row; in Lesson 27 each image is one example.
Introduced in Lesson 26.
- , ,
-
A sequence of inputs , one for each time step . A recurrent network reads them in order.
ExampleIn Lesson 28 each sequence holds bits.
NoteThe subscript numbers time steps, where numbers examples. is a number of steps here, not the test set of Lesson 9 and not a transpose.
Introduced in Lesson 28.
- , ,
-
The vocabulary , a curly V: the fixed list of every token a model can use. is its size, and the tokens have ids . An id is a name, not an amount, so a model turns id into the one-hot vector : numbers, all 0 except a 1 in entry , counting entries from 0 like the ids. A text becomes a sequence of ids , one for each position .
ExampleIn Lesson 29, the vocabulary is space, full stop and the 26 letters, so . The letter a has id 2, so it becomes .
NoteLesson 30’s vocabulary is 27 words and Lesson 33’s the 10 digits. Each merge in Lesson 29 adds one token to . The curly is never the bold of attention. Here the subscript numbers positions in a text, as it numbers time steps in Lesson 28.
Introduced in Lesson 29; used throughout Level 5.
Summaries of a set of values
Centre and spread, the two numbers that sum up a set of values. The examples use the values throughout.
- ,
-
The mean of the values, said “x-bar”: add them up and divide by . is the mean of the values.
Example
NoteIn bold, is the mean point of a set of vectors, worked out one feature at a time (Lesson 18 and Lesson 19). Lesson 18’s is the mean of the points in cluster .
Introduced in Lesson 3. It appears earlier in the maths of Lesson 1.
-
The population mean, said “mu”: the mean of the whole group you care about. When the values are the whole group, .
Example is the mean of feature .
NoteIn Lesson 3’s maths, is also the unknown mean of a larger population that a sample was drawn from. In Lesson 8 it is the mean of . Lesson 13’s and are the means of one class. Lesson 18’s bold is a cluster centre, and Lesson 35’s the mean of a mixture component. In Lesson 33, is the mean of one position’s numbers in layer normalisation.
- ,
-
The population standard deviation and variance, said “sigma”. The variance is the mean squared distance from the mean, dividing by . The SD is its square root, in the units of the data.
Example, so
NoteLesson 7 writes for the population variance of . In Lesson 10, is the variance of the noise , not of the data. In Lesson 3’s maths, is also the unknown variance of a larger population. Lesson 13’s and are the variances of one class, dividing by the class size . Lesson 16’s is how much one tree’s prediction varies from one training set to another. Written with a bracket, is the sigmoid of Lesson 12, not a standard deviation. Lesson 36’s is the standard deviation of the noise added at step .
- ,
-
The sample standard deviation and variance: the same sum of squared distances, divided by instead. They estimate the spread of the bigger group the values were drawn from.
Example, so
Introduced in Lesson 3. Lesson 19 writes and for the sample variances of two features.
- ,
-
The sample covariance matrix of the features, dividing by : , with the centred data . Its diagonal holds each feature’s sample variance, and , off the diagonal, is the sample covariance of features 1 and 2: positive when they tend to rise together.
ExampleFor the three points, .
NoteMany books write the covariance matrix as . These lessons keep for the summation sign.
Introduced in Lesson 19.
-
A z-score: how many standard deviations a value lies from the mean, . Turning every value into its z-score is called standardising.
ExampleFor :
NoteEvery lesson that standardises (Lesson 2, Lesson 3 and Lesson 8) divides by the population SD , so the result has mean 0 and SD 1. From Lesson 12 on, is also a score fed to the sigmoid, and in Level 4 is a layer’s pre-activation; neither is a z-score.
-
The -th smallest value, once the values are sorted. The median is the middle one (or the mean of the middle two when is even).
ExampleWritten in the order 3, 1, 2, the values sort to , so the median is .
Introduced in Lesson 3.
-
Expected value: the long-run average of a random quantity over many repeats of the random draw.
Example: for independent draws, the sample variance equals the variance of the population on average.
NoteLesson 36’s is the mean over a random data point, step and noise draw. The blackboard-bold is not the embedding matrix of Level 5.
Introduced in Lesson 3; used again in Lesson 10 and Lesson 36.
-
The variance of a random quantity: how much it varies from one repeat to the next.
Example is how much the fitted curves disagree at .
Introduced in Lesson 10.
- ,
-
A probability, said “the probability of given ”: how likely class is once the features are known. on its own is the prior, the share of class before looking at the features. A small is a density, the height of a curve such as the normal curve. A density can be above 1; the area under it is a probability.
ExampleBayes’ rule:
Introduced in Lesson 13.
- ,
-
The normal, or Gaussian, distribution with mean and covariance matrix ; in one dimension it is , the bell curve. is its density at . The sign means “is drawn from”. With a vector of zeros and the identity matrix, is a vector of independent standard normal numbers.
ExampleThe standard normal density at its centre is .
NoteLesson 13’s normal curve is the one-dimensional case. Inside the density, is the constant 3.14…, not a mixture weight.
Sample versus population variance
Both average the squared distances from the mean. They differ in what they divide by, because they answer different questions.
- Population variance divides by . It is the spread of exactly these values, treated as the whole group.
- Sample variance divides by . It estimates the spread of the bigger group the values were drawn from. Distances from come out a little too small on average, because sits closer to the values than the unknown population mean does. Dividing by makes up for this: for independent draws, equals the population variance on average.
For , the population variance is about 0.67 and the sample variance is 1. The gap shrinks as grows.
Which lessons use which. Lesson 3 uses the sample version, and , and shows the two side by side. Lesson 19 uses it too: its covariance matrix divides by . Every other variance or standard deviation in the lessons divides by :
- Lesson 2 and Lesson 8 standardise with the population SD, so each standardised feature has SD exactly 1. Lesson 3’s z-scores do the same.
- In Lesson 4, the best constant prediction is the mean, and its MSE is the population variance of .
- Lesson 6 reads and as population variances.
- Lesson 7’s is the population variance of .
- Lesson 10 measures how far its fits spread by dividing by the number of fits, .
- Lesson 11 and Lesson 19 standardise with the population SD, as Lessons 2 and 8 do.
- Lesson 13’s class variances divide by the number of cases in the class, .
- Lesson 17 starts from the mean, and its training MSE is the population variance of .
- Lesson 33’s layer normalisation divides by , the number of entries in one position’s vector: a population variance. Lesson 35’s fitted Gaussians divide by (or by for a mixture component), though its EM start uses the sample covariance, as the lesson says; and Lesson 36 scales its data with the population SD.
Models and their parameters
The numbers a model learns, and the settings that steer the learning. Examples use the three points (1, 1), (2, 3) and (3, 2).
- ,
-
The intercept and slope of a straight line . The intercept is the prediction at ; the slope is how much the prediction changes when goes up by 1.
Example has and .
NoteLesson 8 keeps for the intercept but writes the slope as . From Lesson 12 on, is the intercept instead, as in . Lesson 17’s and are the two levels of a stump.
Introduced in Lesson 1; also in Lesson 4, Lesson 5 and Lesson 6.
- ,
-
The intercept and slope chosen by least squares: the line with the smallest total squared error.
Example
- ,
-
The weights and intercept of a straight boundary between two classes. The score is , and the boundary is the line where it is 0. points across the line, towards the side where the score is positive.
ExampleWith and , the point scores , so it lies on the boundary.
NoteHere is the intercept, not the slope of . With one feature, Lesson 12 writes the score as .
-
The sigmoid: . It squashes any score into a number between 0 and 1, with .
Example, and .
Note here is the sigmoid, not a standard deviation. Each page that uses it says so.
Introduced in Lesson 12. In Level 4 it is one of the activations (Lesson 20).
-
A predicted probability of class 1. Logistic regression gives , and a threshold turns it into a class: when .
ExampleWith , and hours, .
Note is the odds, and its log is the log-odds. In Lesson 15, a plain or is the share of a class in a group of cases, not a prediction.
Introduced in Lesson 12.
- , ,
-
Sums of squared distances from the means, and of products of those distances: . The least-squares slope is .
Example, so .
-
All of a model’s parameters in one vector, said “theta”. is the fitted value.
Example for a straight line.
NoteLesson 8 writes and calls the lowest point . In Lesson 19, a plain is the angle of a line instead, and in Lesson 30 the angle between two vectors. Lesson 35 and Lesson 36 write a plain for a model’s parameters.
Introduced in Lesson 1.
-
Parameter of a polynomial: the number that multiplies . is the intercept.
Example
-
The degree of a polynomial: its highest power of . A degree- polynomial has parameters.
ExampleDegree 3 is .
NoteThe maths of Lesson 8 uses for the number of features instead.
Introduced in Lesson 10.
- ,
-
A function. In Lesson 1, is the model’s rule, with the parameters after the semicolon. In Lesson 5, is any curve whose slope we study. In Lesson 10, is the true curve behind the data and a fitted one.
Example
Introduced in Lesson 1.
-
The average of many fitted curves at one input . It estimates .
ExampleIn Lesson 10 it averages the fits.
Introduced in Lesson 10.
-
A weight: the number that multiplies an input. In these lessons it plays the part of the slope.
ExampleIn Lesson 7, . In Lesson 8, . With features, is the weight on feature .
NoteLessons 1 and 6 call the slope . Lesson 7 fits centred data, and its best equals Lesson 6’s .
Introduced in Lesson 7.
- ,
-
A centred value, said “x-tilde”: the value minus the mean, . Likewise .
Example gives .
Introduced in Lesson 7.
-
The value of at the bottom of the loss: the best one.
Example
-
The loss, written as a function of the parameters: one number that says how badly they fit. Gradient descent tries to make it small.
Example in Lesson 7, in Lesson 8.
NoteIn Lesson 12, is the mean log-loss. In Lesson 17, is the loss of one prediction .
Introduced in Lesson 7.
-
The learning rate, said “eta”: a small positive number that sets the size of each gradient descent step.
ExampleFrom with slope and : .
NoteLesson 12 fits logistic regression with gradient descent, so is the step size there too. In Lesson 17, is the boosting learning rate: each new stump is multiplied by before it is added. It plays the same part, setting the size of each step.
Introduced in Lesson 7; used again in Lesson 8, Lesson 12, Lesson 17 and throughout Level 4, from Lesson 21 on.
-
A count. In gradient descent, the step number: is after steps. In -fold cross-validation, the number of parts the training data is split into. In k-nearest neighbours, the number of neighbours that vote.
Example
NoteIn Lesson 18’s maths, numbers the clusters, as in ; in Lesson 19 it numbers the principal components, as in . The number of clusters is a capital .
Steps: Lesson 7 and Lesson 8. Folds: Lesson 9. Neighbours: Lesson 11.
- ,
-
The training cases nearest to the point : its neighbours, which vote on its class. is the value of that was picked as best.
ExampleWith , holds the three training cases closest to , and the class most of them have wins.
Introduced in Lesson 11.
-
The number of clusters in k-means, a capital letter. You choose it; k-means does not learn it.
Example asks k-means for three groups.
NoteIt is a different thing from the of k-nearest neighbours. In Lesson 14, is a kernel: the dot product of two points after a change of features, worked out straight from the original features.
Introduced in Lesson 18.
- ,
-
is the cluster that point is assigned to, a number from 1 to . is the centre of cluster , a point in the same space as the data.
Example: each point joins its nearest centre.
NoteHere is a centre, not the population mean of the whole data set. The cluster numbers are names, with no order.
Introduced in Lesson 18.
- ,
-
The intercept and weight after standardising : . They map back with and .
Example
Introduced in Lesson 8.
-
The Hessian: the matrix of second derivatives of the loss. It describes how the loss curves in every direction.
Example
Introduced in Lesson 8.
-
The identity matrix: 1s on the diagonal and 0s elsewhere. Multiplying by it changes nothing.
Example
NoteLevel 5 writes it bold, , as in .
Used in the maths of Lesson 8, and in Level 5.
- ,
-
The largest and smallest eigenvalues of the Hessian, said “gamma”: how sharply the loss bowl curves along its steepest and its flattest direction.
ExampleFor standardised data in Lesson 8, , so .
NoteMost books write these eigenvalues as . These lessons keep for the regularisation strength. Lesson 19 writes the eigenvalues of the covariance matrix as : the variances along the two principal components. For the three points, .
-
The condition number, said “kappa”: . It is 1 for a round bowl and large for a long, narrow valley.
Example gives .
Introduced in Lesson 8.
-
The regularisation strength, said “lambda”: how much the penalty on large parameters counts. is plain least squares.
ExampleWith and , that parameter adds to the loss.
NoteIn Lesson 23, is the weight decay strength, in the same role: a penalty on large weights. With plain gradient descent, each step first shrinks every weight by the factor ; with and , .
Introduced in Lesson 10.
-
The penalised loss of ridge regression: the MSE plus times the sum of the squared parameters. The intercept is left out of the penalty.
Example
NoteLesson 23 adds weight decay to a network’s loss, , which penalises the weights but not the biases.
Introduced in Lesson 10.
-
The penalty in a soft-margin support vector machine: how much each point inside the street, or on the wrong side of the line, costs. A large allows few such points; a small one buys a wider street.
ExampleSteps 3 and 4 of Lesson 14 use .
NoteLesson 18’s is the set of points in cluster , not a penalty.
Introduced in Lesson 14.
-
One number per training point in the second form of the support vector problem, the dual, with . The weights are , so only points with , the support vectors, fix the line.
ExampleA point well away from the street has , and moving it a little changes nothing.
Introduced in Lesson 14.
-
The slack of point , said “xi”: how far it is past its own edge of the street, . This is also called the hinge loss.
ExampleA point on or outside its own edge has ; a point exactly on the line has .
Introduced in Lesson 14.
- , ,
-
is the number of trees in a bagged ensemble or random forest. is the class that tree predicts at , and is the set of rows in that tree’s bootstrap sample.
ExampleRow ’s out-of-bag vote uses only the trees with . In a sample of rows drawn with replacement, a given row is left out with probability .
NoteHere numbers the trees; it is not an intercept.
Introduced in Lesson 16.
- ,
-
In gradient boosting, is the model after rounds and is the small tree, a stump, added in round . The start is the mean, and each round adds times a new stump: .
Example after rounds.
Introduced in Lesson 17.
-
The residual of training point at the start of round : . It is the residual of earlier lessons, with . Round fits its stump to these.
ExampleFor the three points, , so .
NoteIt is not the correlation . For the squared loss , it is also minus the gradient of the loss.
Introduced in Lesson 17.
- ,
-
A direction of length 1, for a line at angle . is where centred point lands along it. and are the principal component directions.
ExampleFor the three points, , at 45°.
Introduced in Lesson 19.
- ,
-
Layers of a network are numbered , said “ell”. The hidden layers come first and layer is the output layer. The input counts as layer 0.
ExampleLesson 25’s network has : ten hidden layers and the output layer.
NoteIn Lesson 15, is a leaf of a tree. Earlier lessons write the loss as ; Level 4 writes it .
- ,
-
The weight matrix and bias vector of layer . Row of holds the weights into unit of the layer, so it has one row per unit and one column per input. holds one bias per unit. A layer of units fed by inputs has weights and biases.
ExampleLesson 25’s first layer has 8 units and 2 inputs: weights and biases.
NoteBold capital is a matrix; a plain is one weight. When the output layer has a single unit, its weights form one row, written in Lesson 21, and its bias is one number, . In Lesson 26 and Lesson 27, bold is a filter instead.
Introduced in Lesson 21.
- ,
-
Layer first forms its pre-activation, , then applies the activation to each entry: . The input is layer 0, .
ExampleLesson 22’s network, with sigmoid hidden units, at : and .
NoteLesson 20 has a single unit and writes and , with no layer number. The pre-activation is not a z-score.
Introduced in Lesson 21 (as ) and Lesson 22; in general form in Lesson 25.
- , ,
-
is an activation, applied to each entry on its own. Level 4 uses three: the sigmoid ; tanh, the same S shape running from −1 to 1, ; and , which keeps positive values and turns negative ones into 0.
Example
Note is the activation’s slope, used in backpropagation. here is the sigmoid, not a standard deviation.
Introduced in Lesson 20.
-
The error signal of layer : how much the loss changes as each entry of its pre-activation changes, . Backpropagation finds it from the output backwards, , and each weight’s gradient is the error signal of the unit it feeds times the input it multiplies.
ExampleIn Lesson 22, with a sigmoid output, binary cross-entropy and : .
NoteA layer with one unit has a plain . Lesson 8’s bold is a small move of the parameters, and Lesson 28’s is the error signal at time step .
- ,
-
Momentum keeps a running sum of past gradients and steps along it: , then . , between 0 and 1, sets how long old gradients are remembered.
ExampleIf every gradient were the same , with the sum would grow towards .
NoteLesson 27’s bold holds the pooled values instead.
Introduced in Lesson 24.
- , ,
-
Adam keeps two running means for each weight: of the gradients, with factor , and of the squared gradients, with factor . Both start at 0, so after steps it divides them by and , then steps by , weight by weight. is a tiny number that only guards against dividing by 0. Every lesson here uses , and .
ExampleAt the first step , and dividing by gives back . So the first step moves each weight by about η, whatever the size of its gradient.
NoteLesson 24 writes ; Lesson 27 and Lesson 28 write . Both are this tiny constant, not the noise of Lesson 1. Lesson 24’s RMSProp keeps alone, with factor .
Introduced in Lesson 24; used again in Lesson 23, Lesson 27 and Lesson 28, which train with Adam.
- , ,
-
A convolution slides an filter across an image. The stride is how far the filter moves each time, and the padding is the number of rings of zeros added around the image. The feature map then has rows and columns. Convolution here is cross-correlation (no flip): the filter is laid on the image as written.
ExampleA 12 × 12 image and a 3 × 3 filter: ; with stride 2, ; with padding 1, , the size of the image.
NoteHere is not a sample SD, is not a probability or a polynomial degree, and is not a function.
- , ,
-
In Lesson 27’s network, filter has a 3 × 3 grid of weights and a bias . is its feature map after ReLU and that map after max pooling. The pooled values, in one vector , feed the output unit, which has weights and bias : .
ExampleTwo filters of 9 weights, their 2 biases, 18 output weights and the output bias: .
NoteHere numbers the filters, and is not Lesson 19’s direction.
Introduced in Lesson 27.
- ,
-
The hidden state : what a recurrent network carries from one time step to the next. It starts at and is updated from the new input and the old state, . holds the weights from the old state to the new, the same at every step. and are the input weights and biases, and and turn into the output’s score.
ExampleOne unit with , and , fed then : , then . The 1 is still felt a step later.
NoteBold is a vector of hidden values, not Lesson 5’s small step .
Introduced in Lesson 28.
- , ,
-
The embedding matrix has one row per token, each of width : row , written , is the vector learned for token . Multiplying the one-hot vector by picks out that row, so an embedding is a table look-up. The position embedding has one row per position, and a transformer starts position as , where is the token there.
ExampleIn Lesson 30, 27 words with give numbers, so each word is a point in the plane. Lesson 33 and Lesson 34 use .
NoteLesson 32 writes , with the token as the row number. Lesson 30 writes word ’s row as , and its output matrix has rows . Bold is not the expected value or the number of epochs , and bold is not a probability .
introduced in Lesson 30, in Lesson 32; both used again in Lesson 33 and Lesson 34.
- ,
-
Softmax turns a list of scores into probabilities: . Every output is positive, they add up to 1, and a larger score gets a larger share. The temperature , said “tau”, divides the scores first, : below 1 sharpens the probabilities towards the largest score, above 1 flattens them towards equal shares, and changes nothing.
Example, since . At it is flatter: .
NoteWith two scores, gives for the second, so softmax is the many-class form of the sigmoid of Lesson 12. Lesson 31 reshapes probabilities rather than scores, , which is the same as the softmax of . The letter is because is already a length.
Introduced in Lesson 30; temperature in Lesson 31. Used again in Lesson 32, Lesson 33 and Lesson 34.
- , , , ,
-
Attention. Each position’s vector is mapped to a query , a key and a value , each with entries. Stacked one position per row, they form the matrices , and . A query scores every key by , softmax along each row turns the scores into the attention weights , and each output is the weighted average of the values: . With a causal mask, position may attend only to positions .
ExampleTwo of the keys and values of Lesson 32, and , with and , and the query , so : , then and .
NoteBold holds values; the vocabulary is the curly . Lesson 32 writes the maps as , and , with one position as a column, ; Lesson 33 writes for head and multiplies rows on the right. Bold is not the number of clusters .
Introduced in Lesson 32; used again in Lesson 33 and Lesson 34.
- , ,
-
Multi-head attention runs attention heads side by side, numbered . Each head has its own query, key and value maps and its own weights . Their outputs are placed side by side and mapped back to width by the matrix .
ExampleLesson 33 has and two heads, each of width .
NoteHere counts heads. In Lesson 31 and Lesson 34, is the cross-entropy in bits, and in Lesson 8 it is the Hessian.
- , ,
-
Layer normalisation rescales one position’s numbers to mean 0 and variance 1, then applies a learned scale and shift : . Here and are the mean and the population variance (divide by , not ) of the entries of , and multiplies entry by entry. The tiny constant only guards against dividing by 0.
ExampleFor , with , and left out: , and .
Note and here are vectors learned in training: not momentum , Adam’s and , or the eigenvalues . Lesson 33 uses , a different constant from Adam’s, and writes , and for its three layer norms. Each position is normalised on its own, not across a batch.
- , , ,
-
A generative model learns the data distribution . A mixture of Gaussians writes it as . Component has a weight , a mean and a covariance matrix . The weights are at least 0 and add up to 1: is the share of points the component accounts for (a weight, not 3.14…). In EM, the responsibility is the probability that point came from component , given the current fit; for each point they add up to 1.
ExampleOne dimension, two components with , means 0 and 2 and variance 1. For a point at 0.5, . The two densities differ by a factor , so , with the sigmoid: the point more likely came from the component at 0.
NoteLesson 35 writes for the number of points component accounts for, and fits . Bold is a covariance matrix, not the summation sign; bold is a component’s mean, as it is a cluster centre in Lesson 18. The constant still appears inside the normal density.
Introduced in Lesson 35; the mixture is compared again in Lesson 36.
- , , ,
-
In diffusion, is a data point and the same point after noising steps, . The noise schedule sets how much noise step adds. With , the running product is said “alpha-bar”. After steps, .
ExampleIf , then and , so keeps of .
NoteLesson 36 uses steps, with rising evenly from 0.001 to 0.2. There is a number of steps, not a transpose; is not momentum or a layer norm shift, and is not Lesson 14’s . Lesson 36 writes for the noising recipe, not Lesson 31’s .
Introduced in Lesson 36.
- ,
-
is the noise drawn to make from . The network’s guess of it is , with its weights, and training makes small. To generate, it starts from and steps back to , each step taking out some of the guessed noise and, until the last, adding fresh noise .
ExampleIn two dimensions, always guessing costs on average: each standard normal number has mean square 1.
NoteBold is noise, as is in Lesson 1; it is not the tiny constant of Adam or of layer normalisation. is the standard deviation of the fresh noise ( in Lesson 36), not the sigmoid, and the bold it multiplies is a fresh normal draw, not a pre-activation or a z-score. A plain here holds the network’s weights, as bold does elsewhere.
Introduced in Lesson 36.
Losses and measures of fit
Ways to sum up all the misses in one number. Examples use the least-squares line through (1, 1), (2, 3) and (3, 2), whose residuals are .
-
Mean squared error: the average of the squared residuals, .
Example
NoteLesson 9 and Lesson 10 write and for the MSE on the training and test points.
Introduced in Lesson 4. It appears earlier in the maths of Lesson 1.
-
Mean absolute error: the average size of the residuals, ignoring their sign.
Example
Introduced in Lesson 4.
-
Root mean squared error: the square root of the MSE, back in the units of .
Example
Introduced in Lesson 4.
-
Sum of squared errors: the squared residuals added up, without dividing. .
Example
Introduced in Lesson 6.
-
Total sum of squares: the squared distances of each from , added up. It is the SSE of the flat line .
Example
Introduced in Lesson 6.
-
The share of the variation in that the fitted line accounts for, . For a least-squares fit with an intercept it lies between 0 and 1.
Example
Introduced in Lesson 6.
-
The correlation of and : a number between −1 and 1 that measures how closely the points follow a straight line. For the least-squares line with one input, .
Example, and .
NoteLesson 19 writes for the correlation of two features and for their correlation matrix. Lesson 16’s (rho) is the correlation between two trees’ predictions. Lesson 17’s is a residual, not a correlation.
Introduced in Lesson 6.
- , ,
-
The training set and the test set . Bars around a set mean the number of points in it.
Example
Introduced in Lesson 9.
-
The -fold cross-validation score: the average of the validation MSEs, one for each part held out.
Example, with in Lesson 9.
Introduced in Lesson 9.
-
The number of training sets drawn to estimate bias and variance.
Example
NoteIn Lesson 17, is the number of boosting rounds instead, from 0 to 300 on its slider.
Introduced in Lesson 10.
-
The share of cases a classifier labels correctly: .
Example45 right out of 50: .
Introduced in Lesson 11.
-
The mean log-loss of logistic regression: . Each case is charged minus the log of the probability the model gave to what happened, so a confident mistake costs a lot.
ExampleA pass () given costs .
Introduced in Lesson 12.
- ,
-
With TP, FP and FN the counts of true positives, false positives and false negatives, precision, , is the share of predicted 1s that are right, and recall, , is the share of real 1s that are found.
ExampleWith 8 true positives, 2 false positives and 4 false negatives: and .
Introduced in Lesson 12.
-
The Gini impurity of a group of cases, with the share of class in the group. It is 0 for a pure group; with two classes it is , at most 0.5. A split is scored by , the Gini of its two sides weighted by their sizes.
ExampleThree circles and one triangle: .
Introduced in Lesson 15.
-
The k-means loss: the sum of the squared distances from each point to its own centre, .
ExampleThe three points with one centre at their mean : .
Introduced in Lesson 18.
-
The share of the total variance explained by principal component : .
Example for the three points.
NoteIt is not a residual .
Introduced in Lesson 19.
-
The loss in Level 4, a curly L. Each page says whether it is the loss of one example or the mean over a batch or the training set. Binary cross-entropy is the log-loss of Lesson 12: for one example, .
ExampleA class 0 case given costs : a confident mistake.
NoteLesson 25’s is the mean over a batch; Lesson 28’s is the loss at the last time step alone. In Level 5, is a mean cross-entropy, minus the log of the probability given to the right answer: over word pairs in Lesson 30, steps in Lesson 32, positions in Lesson 33 and windows of text in Lesson 34.
Introduced in Lesson 20.
- ,
-
Cross-entropy in bits per character: the mean of over the characters of a text, where is the probability the model gave the character that really came next. is the log to base 2, so a sure, right guess costs 0 bits and each halving of costs one more bit. Lower is better.
ExampleA uniform guess over the 28 symbols gives every time and costs bits per character.
NoteIt is the many-class form of Lesson 12’s log-loss, in base 2 instead of natural logs: . Lesson 34’s training loss uses natural logs. Here is not the Hessian of Lesson 8 or the number of heads of Lesson 33.
- ,
-
The likelihood of parameters is the density of the data under them, the values taken as independent: . Its log, the log-likelihood , has the same maximum and is easier to work with. Fitting by maximum likelihood makes them as large as possible.
ExampleTwo points with densities 0.5 and 0.2: and .
NoteHere is a likelihood, made large, not a loss , made small, and is not a layer number or a leaf. Lesson 35’s scores are , the mean log-likelihood per point. A density can be above 1, so can have either sign.
Introduced in Lesson 35.
Calculus and other operations
-
Sum, a capital sigma: add up the terms as runs over its range.
Example
Introduced in Lesson 1’s maths.
- ,
-
, a capital pi, multiplies the terms, as adds them. means “proportional to”: equal up to a factor that is the same for every class.
Example. In Lesson 13, .
Introduced in Lesson 13.
-
The indicator: 1 when the statement inside is true and 0 when it is false. Adding it up counts the cases where the statement holds.
Example
Introduced in Lesson 11.
- ,
-
“Is one of” and “is not one of” a set.
Example; says row is not in tree ’s sample.
-
Absolute value: the size of a number, without its sign.
Example
Introduced in Lesson 4.
-
The norm, or length, of a vector: the square root of the sum of its squared entries. The distance between two points is the norm of their difference.
Example
NoteFor a matrix, Lesson 23’s adds up the squares of all its entries. Lesson 25’s is the most can stretch the length of a vector.
-
The derivative of at , said “f-prime” and also written : the slope of the curve there, . The limit is the value the fraction gets as close to as you like as shrinks. Positive means uphill to the right. is the derivative of the derivative.
Example has , so .
NoteLesson 7 writes for the slope of the loss.
Introduced in Lesson 5.
- ,
-
The natural log and its reverse, the exponential, with . , and a log turns a product into a sum. Lesson 13 writes and ; on this site these are the same natural log and exponential.
Example, and .
Introduced in Lesson 12.
-
A partial derivative: the slope as one input changes while the others are held fixed.
Example has .
-
The gradient, said “grad”: all the partial derivatives in one vector. Where it is not zero, it points in the direction in which rises fastest, so gradient descent steps the other way.
Example, which is at .
- ,
-
The dot product: multiply matching entries and add. In Lesson 8, (delta) is a small move of the parameters, and the loss changes by about .
Example
NoteLevel 4’s is something else, a layer’s error signal.
Used in Lesson 8, and from Lesson 12 on, in scores such as .
-
The input that makes something smallest, not the smallest value itself. That value is written .
Example, while .
Note is the input that makes something largest. Lesson 11 picks that way.
Introduced in Lesson 1’s maths; used again in Lesson 9, Lesson 11 and Lesson 18.
-
The transpose: the matrix with its rows turned into columns.
Example
NoteLesson 21, Lesson 22 and Lesson 28 print the raised T upright; it is the same transpose. An italic on the line is a number: the test set of Lesson 9 or the number of time steps in Lesson 28.
Used in the maths of Lesson 8, Lesson 10 and Lesson 19, and in Level 4 from Lesson 21 on.
-
The trace of a square matrix: the sum of the numbers on its diagonal. It equals the sum of the eigenvalues.
Example for the three points.
Introduced in Lesson 19.
-
The entry-by-entry product of two vectors of the same length.
Example
Introduced in Lesson 22; used again in Lesson 25 and Lesson 28.
-
Cross-correlation, which networks call convolution: lay the filter on a patch of the image , multiply each pixel by the weight on top of it and add up, then slide to the next position: .
ExampleLesson 26’s edge filter has columns of , and . On a patch whose right column is three 1s and the rest 0s, .
NoteThe convolution of signal processing flips the filter first. Here it is not flipped.
Introduced in Lesson 26.
-
The floor: round down to a whole number.
Example
Introduced in Lesson 26.
-
Cosine similarity: for two vectors and at angle , . It is 1 when they point the same way, 0 at right angles and −1 when they point opposite ways; their lengths do not matter.
ExampleFor and : , an angle of 45°.
NoteHere is an angle, as in Lesson 19, not a model’s parameters.
Introduced in Lesson 30.
Letters used only in passing
Some letters stand for something only inside one explanation, and are defined where they appear: for a constant being tried out (Lesson 3 and Lesson 4) and for a level of the loss (Lesson 8); and for how far the start is from the bottom along the bowl’s steep and gentle directions (Lesson 8); for how far one value moves (Lesson 3); for a small step along , and as the power in (Lesson 5); and for the heights of the two handles (Lesson 6); for a curve with two dips (Lesson 7); , , and in the arguments about MAE and squaring (Lesson 4); for the factor the distance to the bottom shrinks by at each step, with (bold in Lesson 8) for that distance (Lesson 7 and Lesson 8); for the parameters after steps (Lesson 8); for one chosen input (Lesson 10); for the factor a feature is multiplied by (Lesson 11); for a probability threshold, and TP, FP and FN for counts of right and wrong predictions (Lesson 12); , and for counts of cases, for a class’s log score and for the largest score (Lesson 13); for the tilt of a line, bold for the arrow at right angles to it, for its shift and , for the distances to the nearest point of each class (Lesson 14); and for the feature and threshold of a split, and for the cases sent each way, and for a leaf holding cases (Lesson 15); for the correlation between trees’ predictions (Lesson 16); for a stump’s split point (Lesson 17); and for the points in cluster and how many there are (Lesson 18); and for a point’s distance from a line, for the point rebuilt from one number, and for the correlation matrix (Lesson 19).
In Level 4: for a level of a unit’s score (Lesson 20); for a hidden unit and for the weights into it (Lesson 21); for a small nudge to one weight, and for the input a weight multiplies (Lesson 22); for an epoch, for the epoch with the lowest validation loss, for the gradient on a batch and for the variance of the per-point gradients (Lesson 23); and for positions along and across the valley, for Adam’s step count, and for the grid of learning rates tried, with the best (Lesson 24); and for the numbers of inputs and units of a layer, for the variance of the starting weights, for the diagonal matrix of slopes , and for the block a residual connection skips over (Lesson 25); and for a row and a column, and for a shift (Lesson 26); and for a filter’s map before and after pooling (Lesson 27); and for how many steps back the network must look, for the recurrent layer’s pre-activation, for the change in one hidden state per change in the one before, and , , and for an LSTM’s cell state and gates (Lesson 28).
Several of these clash with letters above:
- Lesson 3’s is a distance moved, and Lessons 7 and 8’s a distance from the bottom, not a number of features.
- Lesson 4’s counts values below ; it is not a polynomial degree.
- Lesson 4’s is a ratio, not a correlation.
- Lesson 4’s is a scale factor, not a step or a fold count.
- Lesson 5’s in is a power, not a number of examples.
- Lesson 6’s and are heights, unlike Lesson 5’s small step .
- Lesson 8’s is a whole parameter vector after steps, not the single parameter .
- Lesson 8’s and are distances, not the constant .
Letters with more than one job
From Level 3 on, a few letters take on a second job. The lesson always says which one it means.
- alone is a standard deviation (Lessons 2 and 3, and Lesson 13). , applied to a score, is the sigmoid of Lesson 12. Each page that uses the sigmoid says so. In Level 4 is again the sigmoid, one of the activations, and each page says so again, as Lesson 32 does. Lesson 36’s is a standard deviation again: that of the fresh noise at step .
- In (Lessons 1 and 6), is the slope. In (Lesson 12, Lesson 13 and Lesson 14) and in , is the intercept. Lesson 16’s numbers the trees, and Lesson 17’s and are the two levels of a stump.
- Small is the number of neighbours in Lesson 11. Capital is the number of clusters in Lesson 18, whose maths uses small only to number the clusters, as in . Lesson 19’s numbers the principal components. Earlier, counted steps (Lessons 7 and 8) and folds (Lesson 9). In Lesson 27, numbers the filters, and in Lesson 28 it is how many steps back the network must look, as again in Lesson 32. In Lesson 31, is the small number added to every count (add- smoothing). Lesson 35’s is the number of mixture components, numbered by . Lesson 14’s is a kernel, not a count.
- is the sample covariance matrix (Lesson 19). The capital sigma is kept for sums, so that covariance matrix is not written . The one exception is Lesson 35’s bold , the covariance matrix of a Gaussian, written as most books write it. Lesson 16’s is a set of rows, and Lesson 6’s a sum of squares.
- is a correlation (Lesson 6 and Lesson 19). is a boosting residual (Lesson 17): the residual at round . Lesson 4’s is a ratio. Lesson 29’s counts merges, Lesson 30’s is the length of a vector, and Lesson 35’s is a responsibility.
- is the number of training sets drawn in Lesson 10, and the number of boosting rounds in Lesson 17.
- is the number of features (Lesson 2 and Lesson 13). In Lesson 15 it is the depth of a tree. Lesson 14’s and , and Lesson 19’s , are distances. In Level 5, is the embedding width, the number of entries in each token’s vector (Lesson 30, Lesson 33 and Lesson 34), and the width of a query or key (Lesson 32 and Lesson 33).
- Class labels are , except in Lesson 14, which uses so that one inequality covers both classes.
- is the SVM penalty (Lesson 14). Lesson 18’s is the set of points in cluster , and is the cluster of point , not a constant . In Lesson 29 and Lesson 31, is a count, and Lesson 30’s is the centre word.
- numbers the boosting rounds in Lesson 17; in Lesson 13 it is the largest log score. Lesson 24’s bold is a running sum or mean of gradients, and Lesson 27’s bold the pooled values. Lesson 33’s is the width of the small network in each block, and in Lesson 35 and Lesson 36, is the number of samples.
- holds a model’s parameters; in Lesson 19, is the angle of a line, and in Lesson 30 the angle between two vectors. Lesson 35 and Lesson 36 write a plain for a model’s parameters.
- Lesson 19’s is a share of variance explained, not a residual . Its is a correlation matrix, not the training set of Lesson 9.
- Lesson 14’s bold is an arrow at right angles to a line, not the number of examples .
- is a probability threshold in Lesson 12 and a split threshold in Lesson 15. Lesson 19’s is a position along a line.
- Lesson 15’s and are shares of a class in a group of cases, not a predicted probability or a polynomial degree.
- Bold (Level 4) is the bias vector of layer , one bias per unit; a plain or is the bias of a single unit. It does the job of the intercept of Lesson 12, not the slope of Lessons 1 and 6. Lesson 27’s and are a filter’s and the output’s biases, and Lesson 28’s and the recurrent layer’s and the output’s.
- is the strength of the ridge penalty in Lesson 10 and of weight decay in Lesson 23: the same role, a penalty on large weights. Eigenvalues are written so that keeps this one job.
- is the number of trees in Lesson 16 and a band of feature values in Lesson 13. In Lesson 23, is a batch of training points and the number of points in it; Lesson 13’s counts the cases in its band.
- is the test set in Lesson 9 and the number of time steps in Lesson 28. A transpose is a raised mark, as in , never a on the line. In Level 5, is the length of a sequence (Lesson 32, Lesson 33 and Lesson 34) or the number of noising steps (Lesson 36).
- Bold capital is a weight matrix, as in , or a filter in Lesson 26 and Lesson 27. A plain is a single weight and bold a vector of weights. Lesson 28’s is the matrix from the old hidden state to the new.
- is the padding in Lesson 26. It is not a predicted probability , the polynomial degree of Lesson 10 or a share of a class as in Lesson 15.
- is the stride in Lesson 26 and the sample SD in Lesson 3; Lesson 17’s is a split point. Lesson 24’s bold is a running mean of squared gradients. Lesson 32’s is an attention score and a position. Lesson 33’s is the population variance of one position’s numbers, not the sample variance.
- Adam’s (Lesson 24), written in Lesson 27 and Lesson 28, is a tiny constant that guards against dividing by 0. It is not the noise of Lesson 1.
- is the number of layers in Lesson 25. Lesson 7, Lesson 8, Lesson 12 and Lesson 17 write a loss as ; Level 4 writes it . Lesson 35’s is a likelihood, made large, and its the log-likelihood, not a layer.
- is a function in Lessons 1, 5 and 10 and the filter size in Lesson 26. Lesson 25’s is the block a residual connection skips over, and Lesson 28’s is an LSTM gate.
- is a small step in Lesson 5 and Lesson 22. Bold in Lesson 28 is a hidden state.
- is the variance of the starting weights in Lesson 25 and a position across the valley in Lesson 24. Bold is a direction in Lesson 19 and the output weights in Lesson 27.
- Bold is the matrix of values in attention (Lesson 32 to Lesson 34). The vocabulary is the curly (Lesson 29 on), never .
- is the momentum factor in Lesson 24, and , are Adam’s. Lesson 33’s bold is the learned shift of layer normalisation, and Lesson 36’s the noise schedule.
- is an eigenvalue in Lesson 8 and Lesson 19. Lesson 33’s bold is the learned scale of layer normalisation.
- is a mixture weight in Lesson 35, not 3.14…. A plain is still the constant, as in the normal density and Lesson 36’s time features . Lesson 33’s bold capital is a permutation matrix, a shuffle of the positions; it has nothing to do with or the constant .
- is the temperature (Lesson 31 and Lesson 34). It is a Greek tau because is already a length or a number of steps.
- Bold is the embedding matrix (Level 5). Lesson 23’s is the number of epochs, and the blackboard-bold is an expected value (Lesson 3 and Lesson 36).
- is the Hessian in Lesson 8, the cross-entropy in bits in Lesson 31 and Lesson 34, and the number of attention heads in Lesson 33.
- Bold in Lesson 36 is noise, a vector drawn from , like the noise of Lesson 1. The in Lesson 33’s layer normalisation is a tiny constant, like Adam’s.
- Bold is a query (Lesson 32). Lesson 31’s is a set of probabilities reshaped by temperature, and Lesson 36’s is the noising recipe.
- In Level 5, counts tokens or scored positions (Lesson 29, Lesson 31 and Lesson 34) or word pairs (Lesson 30); is still the number of examples. Lesson 35’s is the number of points a mixture component accounts for.