Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 3 · Classic ML

Principal component analysis

Find the direction in which the data vary most, and keep most of the information with fewer numbers.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

k-means looked for groups without any labels. Principal component analysis, PCA for short, also works without labels, but it looks for directions. It asks: along which direction do the points spread out the most?

We centre the data, turn a line through it, find the best direction, add the second one, squeeze each person down to one number, and finish with what goes wrong when the units change.

The people are made up. See "About the data" at the end.

You'll learn

  • why the direction of greatest spread is the top eigenvector of the covariance matrix
  • how much of the spread one number per person keeps, and what is lost
  • why PCA depends on the units, and how standardising fixes that
  1. Centre the data

    PCA is about how the points spread around their centre. So first, subtract the means.

    Each dot is one person: height across, arm span up, both in centimetres. Taller people tend to have longer arms, so the cloud leans. There are 50 people.

    The cloud keeps its shape. Only the place we call zero moves. Use Raw and Centred to compare the two sets of axes.

    The maths

    Write person ii as xi=(xi1,xi2)\mathbf{x}_i = (x_{i1}, x_{i2}) and the mean as xˉ\bar{\mathbf{x}}. The centred point is

    x~i=xi−xˉ,xˉ=1n∑i=1nxi\tilde{\mathbf{x}}_i = \mathbf{x}_i - \bar{\mathbf{x}}, \qquad \bar{\mathbf{x}} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{x}_i

    Stack the centred points as the rows of an n×2n \times 2 matrix X~\tilde{X}. Each of its columns adds up to 0.

  2. Spin a line

    Turn a line through the centre and drop each point onto it at a right angle. The spread along the line and the spread across it trade off.

    The small cyan dots are where each person lands on the line. The grey segments join each person to that spot, at right angles to the line. Use the angle slider to turn it.

    Along: square each cyan dot's distance from the centre, add them up and divide by n − 1. That is the sample variance of the positions along the line (Lesson 3 explains n − 1). Across: do the same with the lengths of the grey segments.

    The maths

    A line through the centre at angle θ\theta has the unit direction v=(cos⁡θ,sin⁡θ)\mathbf{v} = (\cos\theta, \sin\theta). Person ii lands at position ti=x~i⋅vt_i = \tilde{\mathbf{x}}_i \cdot \mathbf{v} along it, and sits a distance did_i from it. By Pythagoras, ∥x~i∥2=ti2+di2\lVert\tilde{\mathbf{x}}_i\rVert^2 = t_i^2 + d_i^2. Add up over everyone and divide by n−1n - 1:

    1n−1∑iti2⏟along+1n−1∑idi2⏟across=s12+s22\underbrace{\frac{1}{n-1}\sum_i t_i^2}_{\text{along}} + \underbrace{\frac{1}{n-1}\sum_i d_i^2}_{\text{across}} = s_1^2 + s_2^2

    where s12s_1^2 and s22s_2^2 are the sample variances of height and arm span. The right side does not depend on θ\theta.

    Both parts here divide by n−1n - 1. The mean squared distance to the line divides by nn instead, so with it the identity needs a factor: along+nn−1⋅1n∑idi2=s12+s22\text{along} + \frac{n}{n-1}\cdot\frac{1}{n}\sum_i d_i^2 = s_1^2 + s_2^2.

  3. The best direction

    The direction with the most spread along it is the first principal component, PC1. We do not have to search for it: it can be computed.

    PC1 is the eigenvector of S with the larger eigenvalue. An eigenvector is a direction that S only stretches, without turning it: Sv = γv. The stretch, γ, is its eigenvalue. The loss surface lesson met eigenvalues too, as the curvatures of its bowl.

    A direction has no arrowhead: −v₁ gives the same line. Software may hand you either sign. This page always makes the larger of its two numbers (in size) positive.

    The maths

    The covariance matrix, dividing by n−1n - 1 (the sample covariance), is

    S=1n−1X~⊤X~=(s12s12s12s22)S = \frac{1}{n-1}\tilde{X}^{\top}\tilde{X} = \begin{pmatrix} s_1^2 & s_{12} \\ s_{12} & s_2^2 \end{pmatrix}

    We write it SS, not Σ\Sigma, because Σ\Sigma is the summation sign on this site. The variance along a unit direction v\mathbf{v} is

    1n−1∑i(x~i⋅v)2=v⊤S v\frac{1}{n-1}\sum_i (\tilde{\mathbf{x}}_i \cdot \mathbf{v})^2 = \mathbf{v}^{\top} S\,\mathbf{v}

    Maximise it subject to ∥v∥=1\lVert\mathbf{v}\rVert = 1. Setting the gradient of v⊤Sv−γ(v⊤v−1)\mathbf{v}^{\top}S\mathbf{v} - \gamma(\mathbf{v}^{\top}\mathbf{v} - 1) to zero gives

    Sv=γ v,v⊤S v=γS\mathbf{v} = \gamma\,\mathbf{v}, \qquad \mathbf{v}^{\top}S\,\mathbf{v} = \gamma

    So the best direction is an eigenvector, and the variance along it is its eigenvalue: the largest one, γ1\gamma_1. For a 2 × 2 matrix the two eigenvalues, γ1≥γ2\gamma_1 \ge \gamma_2, are

    γ1,2=s12+s222±(s12−s222)2+s122\gamma_{1,2} = \frac{s_1^2 + s_2^2}{2} \pm \sqrt{\left(\frac{s_1^2 - s_2^2}{2}\right)^2 + s_{12}^2}

    Most books write eigenvalues as λ\lambda. This explorer keeps λ\lambda for the regularisation strength, so here they are γ\gamma.

  4. PC2, and the share of the variance

    The second principal component, PC2, is at right angles to PC1. Together they split the total spread into two parts.

    The arrows are two standard deviations long along each direction: 2√γ₁ for PC1, solid, and 2√γ₂ for PC2, dashed.

    The maths

    The sum of the eigenvalues of a matrix equals the sum of its diagonal, the trace:

    γ1+γ2=tr⁡S=s12+s22,ek=γkγ1+γ2\gamma_1 + \gamma_2 = \operatorname{tr} S = s_1^2 + s_2^2, \qquad e_k = \frac{\gamma_k}{\gamma_1 + \gamma_2}

    Here eke_k is the share of the variance explained by PCkk; ee is a share, not a residual. Because SS is symmetric, its eigenvectors are at right angles. Measured along v1\mathbf{v}_1 and v2\mathbf{v}_2, the two new coordinates have variances γ1\gamma_1 and γ2\gamma_2 and covariance 0.

    With many features, software finds the same directions with the singular value decomposition (SVD) of X~\tilde{X}, without forming SS first. We do not need it here.

  5. Keep one number per person

    Keep only each person's position along PC1: one number instead of two. Then rebuild both measurements from that number.

    A person's score is their position along PC1, measured from the centre. The panel shows all 50 scores on one line.

    To rebuild a person, start at the mean and go their score along v₁. Rebuilt moves every point onto the PC1 line. The grey segments are what is lost.

    The maths

    The score and the rebuilt point are

    ti=x~i⋅v1,x^i=xˉ+ti v1t_i = \tilde{\mathbf{x}}_i \cdot \mathbf{v}_1, \qquad \hat{\mathbf{x}}_i = \bar{\mathbf{x}} + t_i\,\mathbf{v}_1

    The lost part, xi−x^i\mathbf{x}_i - \hat{\mathbf{x}}_i, runs at right angles to the PC1 line; its length is the point's distance from the line, which is the size of its PC2 coordinate. So, with the nn versus n−1n - 1 factor written out,

    1n∑i∥xi−x^i∥2=n−1n γ2\frac{1}{n}\sum_i \lVert\mathbf{x}_i - \hat{\mathbf{x}}_i\rVert^2 = \frac{n-1}{n}\,\gamma_2

    By the identity in step 2, the line with the most variance along it has the least across it. So PC1 is also the line through the centre closest to the points, in mean squared distance. With more features, keeping the first few scores works the same way.

  6. Units matter

    Write arm span in millimetres and PCA gives a different answer. Standardising first takes the units out.

    So should you always standardise? If the features share a unit and their spreads matter, like two lengths in cm, plain PCA can be the right choice. If the units differ or are arbitrary, standardise.

    The maths

    Standardising uses zij=(xij−xˉj)/σjz_{ij} = (x_{ij} - \bar{x}_j)/\sigma_j, with the population SD σj\sigma_j (divide by nn). The sample covariance of the z-scores is then

    Sz=nn−1 R,R=(1rr1)S_z = \frac{n}{n-1}\,R, \qquad R = \begin{pmatrix} 1 & r \\ r & 1 \end{pmatrix}

    where RR is the correlation matrix. Multiplying a matrix by a number does not change its eigenvectors or the shares, so the choice of nn or n−1n - 1 here does not change the answer. The eigenvalues of RR are 1±r1 \pm r, with eigenvectors along (1,1)(1, 1) and (1,−1)(1, -1). That is why PC1 sits at 45° and its share is (1+r)/2(1 + r)/2 (for r>0r > 0; if r<0r < 0, PC1 is along (1,−1)(1, -1), at 135°, with share (1−r)/2(1 - r)/2; if r=0r = 0, every direction ties).

    Multiplying arm span by 10 multiplies s22s_2^2 by 100 and s12s_{12} by 10, but leaves rr unchanged. So plain PCA moves and standardised PCA does not.

Quick check

What does PC1 maximise?

Right. Of all directions through the centre, PC1 has the most spread along it, and that spread is the eigenvalue γ₁.

The other way round. Along and across add up to a fixed total, so the direction with the most variance along it has the least distance across it. PC1 minimises the mean squared distance.

PCA does not change the data, so the correlation stays what it is. PCA only chooses a direction to look along.

Hardly any point lies exactly on any line. PCA works with squared distances and variances, not with counts.

After switching arm span to mm, PC1 points where, and why?

Right. PC1 sits at °, only ° from the arm-span axis. Multiplying arm span by 10 multiplies its variance by 100, and PC1 follows the variance.

The people are the same, but PCA sees only numbers. Bigger numbers mean a bigger variance, and PC1 follows the variance.

PC1 follows the larger variance, not the smaller one, and precision plays no part. Height now has by far the smaller variance.

Plain PCA does not rescale anything. PC1 lands at 45° only after you standardise both features yourself.

Lesson complete

  • PCA centres the data, then finds the direction with the most variance along it: PC1.
  • PC1 is the eigenvector of the covariance matrix S with the largest eigenvalue γ₁, and γ₁ is the variance along it.
  • The shares γₖ ÷ (γ₁ + γ₂) say how much of the total variance each direction carries; they add up to 1.
  • Keeping only the PC1 score loses the least, in mean squared distance, of any single line through the centre.
  • PCA depends on the units. Standardise when the features are measured in different or arbitrary units.

Common misconceptions

  • "PC1 is the regression line of arm span on height." Regression makes the up-and-down distances small and treats the two features differently. PCA makes the right-angle distances small and treats both features alike, so the two lines differ.
  • "A large share means PC1 is good for predicting an outcome." PCA never sees an outcome. The direction with the most spread need not be the one that matters for a prediction.
  • "The signs of PC1 mean something." A direction and its opposite give the same line. Software may flip the sign; the line stays put.
Next lesson: The artificial neuron →

Further reading

  1. James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 12.2, "Principal Components Analysis". It covers the maximum-variance view (12.2.1), the closest-line view (12.2.2), the proportion of variance explained (12.2.3) and "Scaling the Variables" (12.2.4). Its variances divide by n, so its version of the step 2 identity has no n ÷ (n − 1) factor.
  2. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Chapter 10, "Dimensionality Reduction with Principal Component Analysis", for the linear algebra in full.

About the data. The 50 people are made up. Height = 170 + 8 z₁ cm and arm span = 0.9 × (height − 170) + 170 + 4 z₂ cm, where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1, made with mulberry32, seed 111, drawn z₁ then z₂ for each person in turn. Step 6 uses the same people with arm span × 10, in mm. The ringed person is the first one drawn. Variances and covariances divide by n − 1 = 49 (sample), the mean squared error in step 5 divides by n = 50, and standardising uses the population SD (divide by n). Displayed values are rounded; the calculations are not.

If this helped, you can .