Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 1 · Data & intuition

Data as numbers

A model only ever sees numbers. Each example becomes a short list of numbers, a vector, which is also a point you can draw and measure.

About 10 minutes · No maths needed; the maths is there if you want it

Overview

Imagine six pieces of fruit. Each one has been weighed and measured across, and the results written in a table. Before a model can learn anything from them, the fruit have to become numbers like these.

Once each fruit is a list of numbers, we can draw it as a point and measure how far apart two fruit are. Many methods use that distance to decide which examples are alike. There is a catch: the answer can depend on the units you measured in.

The fruit in this lesson are made up. See "About the data" at the end.

You'll learn

  • what examples, features and labels are
  • how one example becomes a vector, and a point on a graph
  • how to measure the distance between two examples, and why units can change the answer
  1. Rows and columns

    Each row of the table is one fruit, called an example. Each column is one thing measured about it, called a feature.

    There are six examples and two features: mass in grams, and diameter in centimetres. The diameter is the width across the widest part.

    The kind column is different. It is the answer a model might later learn to predict from the measurements, like the mark in Lesson 1. A column like this is called the label. It is what a model would predict, so it is not one of the features.

    The six fruit
    #kindmass (g)diameter (cm)
    1apple1407.0
    2orange2308.0
    3nectarine1366.4
    4plum625.0
    5apple1527.1
    6apricot404.4
    The maths

    We write nn for the number of examples and dd for the number of features. Here n=6n = 6 and d=2d = 2.

    The feature columns together form a grid of numbers with nn rows and dd columns. It is called the data matrix and written X\mathbf{X}.

  2. One row is a vector

    Take one row and keep only its features. You get an ordered list of two numbers, called a vector, written x = (x₁, x₂).

    Here x₁ is the mass and x₂ is the diameter. The order is fixed: mass always comes first, so every fruit's vector reads the same way.

    Vectors are written in bold. So x stands for a whole row, while x₁ and x₂ are single numbers inside it.

    Pick a fruit from the menu to see its vector.

    The maths

    For example number ii,

    xi=(xi1, xi2)∈R2\mathbf{x}_i = (x_{i1},\, x_{i2}) \in \mathbb{R}^2

    The first subscript says which example, and the second says which feature. R2\mathbb{R}^2 is the set of all pairs of real numbers.

    With dd features, a vector has dd entries: x=(x1,…,xd)\mathbf{x} = (x_1, \dots, x_d).

  3. A vector is a point

    Two numbers give a place on a graph: go x₁ across and x₂ up. Each fruit becomes a point.

    The table has turned into a picture. Across is mass, up is diameter, and each dot is one fruit, numbered as in the table.

    The arrow runs from the corner, where both values are zero, to the chosen fruit. It is the same vector drawn as a movement. The dashed lines show its two parts.

    Heavier fruit sit further right, and wider fruit higher up. In these fruit the two go together, so the dots run from bottom left to top right.

    The maths

    The arrow starts at the origin, 0=(0,0)\mathbf{0} = (0, 0). Its two parts are the short sides of a right-angled triangle, and the arrow is the long side. So, by Pythagoras' theorem, its length is

    ∥x∥=x12+x22\lVert \mathbf{x} \rVert = \sqrt{x_1^2 + x_2^2}

    This length is called the norm of x\mathbf{x}.

  4. Distance between two fruit

    Pick two fruit, a and b. The distance between their points measures how different they are.

    The dashed sides are the gap in mass and the gap in diameter. The solid line is the long side of that right-angled triangle. Its length comes from Pythagoras: square both gaps, add them, and take the square root.

    A small distance means two fruit have similar measurements. Many methods use this to find examples that are alike.

    The two axes use different scales, so the length of the line on screen is not the distance. The number in the box is.

    Notice that the distance adds a gap in grams to a gap in centimetres. Keep that in mind for the next step.

    The maths

    The distance between a\mathbf{a} and b\mathbf{b} is the norm of their difference, a−b\mathbf{a} - \mathbf{b}:

    ∥a−b∥=(a1−b1)2+(a2−b2)2\lVert \mathbf{a} - \mathbf{b} \rVert = \sqrt{(a_1 - b_1)^2 + (a_2 - b_2)^2}

    With dd features it is ∑k=1d(ak−bk)2\sqrt{\sum_{k=1}^{d} (a_k - b_k)^2}. This is the Euclidean distance. It is the same whichever fruit you call a, and it is zero only when the two vectors are equal.

  5. Units change the answer

    In the distance, a gap of 1 g counts as much as a gap of 1 cm. But 1 g is a tiny difference between two fruit, and 1 cm is a big one.

    In grams and centimetres, mass decides almost everything. Fruit of about the same weight come out close, even when their sizes differ.

    Standardising fixes this. For each feature, subtract its mean (the average) and divide by its standard deviation (a measure of how spread out the values are). Each feature is then counted in standard deviations from its average, and has no units, as long as the feature's values are not all the same.

    Switch between Raw units and Standardised to compare. In the standardised view both axes use the same scale, so the length on screen does match the distance.

    Had we weighed in kilograms, the diameters would decide almost everything instead. Standardising gives the same result whatever units you start from.

    Standardising is one common choice, not the only one. Which scale is right depends on the problem.

    The maths

    Write xijx_{ij} for the value of feature jj in example ii, as in step 2. Each feature jj has its own mean μj\mu_j and standard deviation σj\sigma_j, taken over the nn examples. The standardised value, or z-score, is

    zij=xij−μjσjz_{ij} = \frac{x_{ij} - \mu_j}{\sigma_j}

    where

    μj=1n∑i=1nxij\mu_j = \frac{1}{n}\sum_{i=1}^{n} x_{ij}
    σj=1n∑i=1n(xij−μj)2\sigma_j = \sqrt{\frac{1}{n}\sum_{i=1}^{n} (x_{ij} - \mu_j)^2}

    This page uses the population standard deviation, which divides by nn. Lesson 3 explains standard deviation properly, and when to divide by n−1n - 1 instead. If a feature's values were all the same, σj\sigma_j would be 0 and the z-score would not be defined.

    Afterwards each feature has mean 0 and population standard deviation 1. Changing units, say from grams to kilograms, multiplies every xijx_{ij}, μj\mu_j and σj\sigma_j for that feature by the same number, so zijz_{ij} does not change.

Quick check

In raw units (grams and centimetres), which two fruit are closest?

Right. They have the smallest gap in mass of any pair, and in raw units the gram gap decides almost everything. Their diameters differ by more than the two apples' do, but that barely counts.

The two apples come out closest only after standardising. In raw units their masses differ by more grams than fruit 1 and 3, and grams decide.

The plum and the apricot are similar in size, but their gap in mass is bigger than that of fruit 1 and 3. In raw units that puts them further apart.

The nectarine and fruit 5 differ by more grams than fruit 1 and 3, so in raw units they are further apart.

After standardising both features, which two fruit are closest?

They were closest in raw units. After standardising, their gap in diameter counts for much more than their gap in mass, and it pushes them apart.

Right. Once each feature is measured against its own spread, the two apples are close in both mass and diameter, so they are the nearest pair.

The plum and the apricot are fairly close, but the two apples are closer in both mass and diameter.

The nectarine is narrower than both apples, and after standardising that difference counts. Fruit 1 and 5 are closer.

Lesson complete

  • A data table has one row per example and one column per feature.
  • Each example is a vector: its feature values in a fixed order. A vector is also a point.
  • Distance compares examples, but it is only fair once the features are on comparable scales.

Common misconceptions

  • "A vector has to be an arrow, as in physics." Here it is simply an ordered list of numbers. The arrow is one way to draw it.
  • "The length on the graph is the distance." Only when both axes use the same scale.
  • "Numbers are numbers, so units don't matter." Switching from grams to kilograms can change which examples count as close.
  • "Standardising throws information away." It shifts and rescales each feature. The values keep their order and their relative gaps within each feature. Keep each feature's mean and standard deviation and you can undo it.
Next lesson: Centre and spread →

Further reading

  1. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 8.1.1, "Data as Vectors", and Section 3.3, "Lengths and Distances".
  2. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 4.7.6, "K-Nearest Neighbors", which explains why the scale of the variables matters for distances.

About the data. The six fruit are made up. Their masses and diameters are realistic but invented for this lesson. They are fixed values written into the page, not random draws, so there is no seed. Treated as a sphere, each fruit's mass and diameter give a density between about 0.8 and 1 gram per cubic centimetre, close to that of water, so the two columns agree with each other.

If this helped, you can .