ML Explorer · Level 3 · Classic ML
Principal component analysis
Find the direction in which the data vary most, and keep most of the information with fewer numbers.
Overview
k-means looked for groups without any labels. Principal component analysis, PCA for short, also works without labels, but it looks for directions. It asks: along which direction do the points spread out the most?
We centre the data, turn a line through it, find the best direction, add the second one, squeeze each person down to one number, and finish with what goes wrong when the units change.
The people are made up. See "About the data" at the end.
You'll learn
- why the direction of greatest spread is the top eigenvector of the covariance matrix
- how much of the spread one number per person keeps, and what is lost
- why PCA depends on the units, and how standardising fixes that
-
Centre the data
PCA is about how the points spread around their centre. So first, subtract the means.
Each dot is one person: height across, arm span up, both in centimetres. Taller people tend to have longer arms, so the cloud leans. There are 50 people.
The mean height is cm and the mean arm span is cm. Subtract them from every person, and the centre of the cloud sits at (0, 0).
The ringed person measures (, ) cm. Centred, they are (, ): a little taller than average, with arms well short of average.
The cloud keeps its shape. Only the place we call zero moves. Use Raw and Centred to compare the two sets of axes.
The maths
Write person as and the mean as . The centred point is
Stack the centred points as the rows of an matrix . Each of its columns adds up to 0.
-
Spin a line
Turn a line through the centre and drop each point onto it at a right angle. The spread along the line and the spread across it trade off.
The small cyan dots are where each person lands on the line. The grey segments join each person to that spot, at right angles to the line. Use the angle slider to turn it.
Along: square each cyan dot's distance from the centre, add them up and divide by n − 1. That is the sample variance of the positions along the line (Lesson 3 explains n − 1). Across: do the same with the lengths of the grey segments.
At 0° the line is the height axis. Along is then the variance of height, cm², and across is the variance of arm span, cm². Together they make cm².
That total, cm², is the same at every angle. Whatever spread the line gains along it, it loses across it. The chart shows the two curves, with the total as a dotted line.
The maths
A line through the centre at angle has the unit direction . Person lands at position along it, and sits a distance from it. By Pythagoras, . Add up over everyone and divide by :
where and are the sample variances of height and arm span. The right side does not depend on .
Both parts here divide by . The mean squared distance to the line divides by instead, so with it the identity needs a factor: .
-
The best direction
The direction with the most spread along it is the first principal component, PC1. We do not have to search for it: it can be computed.
On the slider, the best whole degree is °. The exact best direction is °, and Snap to PC1 turns the line there. The variance along it is cm², more than at any other angle. We write it γ₁ (gamma one).
Going 1 cm along PC1 means going cm in height and cm in arm span. That pair is the direction v₁.
Where does it come from? Put the two variances and the covariance, all dividing by n − 1, into a 2 × 2 table called the covariance matrix, S. Height's variance is , arm span's is , and their covariance is , all in cm².
PC1 is the eigenvector of S with the larger eigenvalue. An eigenvector is a direction that S only stretches, without turning it: Sv = γv. The stretch, γ, is its eigenvalue. The loss surface lesson met eigenvalues too, as the curvatures of its bowl.
A direction has no arrowhead: −v₁ gives the same line. Software may hand you either sign. This page always makes the larger of its two numbers (in size) positive.
The maths
The covariance matrix, dividing by (the sample covariance), is
We write it , not , because is the summation sign on this site. The variance along a unit direction is
Maximise it subject to . Setting the gradient of to zero gives
So the best direction is an eigenvector, and the variance along it is its eigenvalue: the largest one, . For a 2 × 2 matrix the two eigenvalues, , are
Most books write eigenvalues as . This explorer keeps for the regularisation strength, so here they are .
-
PC2, and the share of the variance
The second principal component, PC2, is at right angles to PC1. Together they split the total spread into two parts.
PC2 is the other eigenvector of S. With two features it is simply the direction at right angles to PC1, v₂ = (, ). The variance along it is γ₂ = cm², the least of any direction: it is the across value when the line sat on PC1.
The two eigenvalues add up to the total: + = cm². So PC1's share of the variance is γ₁ ÷ (γ₁ + γ₂) = , and PC2's is . The two shares always add up to 1.
The arrows are two standard deviations long along each direction: 2√γ₁ for PC1, solid, and 2√γ₂ for PC2, dashed.
The maths
The sum of the eigenvalues of a matrix equals the sum of its diagonal, the trace:
Here is the share of the variance explained by PC; is a share, not a residual. Because is symmetric, its eigenvectors are at right angles. Measured along and , the two new coordinates have variances and and covariance 0.
With many features, software finds the same directions with the singular value decomposition (SVD) of , without forming first. We do not need it here.
-
Keep one number per person
Keep only each person's position along PC1: one number instead of two. Then rebuild both measurements from that number.
A person's score is their position along PC1, measured from the centre. The panel shows all 50 scores on one line.
To rebuild a person, start at the mean and go their score along v₁. Rebuilt moves every point onto the PC1 line. The grey segments are what is lost.
The ringed person, (, ) cm, has score . Rebuilt, they are (, ) cm, cm away. Their unusually short arms are lost: one number can only say where someone sits along the main trend.
Over all 50 people, the mean squared length of the grey segments is cm². This is a mean, so it divides by n = 50, and it equals (49 ÷ 50) × γ₂. Here γ₂ = cm² is step 2's across value with the line on PC1, which divides by n − 1 instead. No other line through the centre loses less. One number per person keeps of the variance.
The maths
The score and the rebuilt point are
The lost part, , runs at right angles to the PC1 line; its length is the point's distance from the line, which is the size of its PC2 coordinate. So, with the versus factor written out,
By the identity in step 2, the line with the most variance along it has the least across it. So PC1 is also the line through the centre closest to the points, in mean squared distance. With more features, keeping the first few scores works the same way.
-
Units matter
Write arm span in millimetres and PCA gives a different answer. Standardising first takes the units out.
These are the same 50 people, with arm span now in mm. Every arm-span number is 10 times larger, so its variance is 100 times larger: mm² against height's cm². PCA sees only the numbers, not the units.
Arm span now carries almost all of the variance. PC1 swings to °, only ° from the arm-span axis, with a share of . Both axes use the same number scale, so the cloud is a thin upright streak.
The fix is to standardise each feature first: subtract its mean and divide by its standard deviation, here the population SD (dividing by n). Then cm or mm makes no difference: PC1 is at ° either way, with a share of .
With exactly two standardised features, the answer is fixed by the correlation r, here . When they rise together (r > 0), PC1 lies at 45° and its share is (1 + r) ÷ 2. With more features the direction is not fixed in advance.
So should you always standardise? If the features share a unit and their spreads matter, like two lengths in cm, plain PCA can be the right choice. If the units differ or are arbitrary, standardise.
The maths
Standardising uses , with the population SD (divide by ). The sample covariance of the z-scores is then
where is the correlation matrix. Multiplying a matrix by a number does not change its eigenvectors or the shares, so the choice of or here does not change the answer. The eigenvalues of are , with eigenvectors along and . That is why PC1 sits at 45° and its share is (for ; if , PC1 is along , at 135°, with share ; if , every direction ties).
Multiplying arm span by 10 multiplies by 100 and by 10, but leaves unchanged. So plain PCA moves and standardised PCA does not.
Quick check
Lesson complete
- PCA centres the data, then finds the direction with the most variance along it: PC1.
- PC1 is the eigenvector of the covariance matrix S with the largest eigenvalue γ₁, and γ₁ is the variance along it.
- The shares γₖ ÷ (γ₁ + γ₂) say how much of the total variance each direction carries; they add up to 1.
- Keeping only the PC1 score loses the least, in mean squared distance, of any single line through the centre.
- PCA depends on the units. Standardise when the features are measured in different or arbitrary units.
Common misconceptions
- "PC1 is the regression line of arm span on height." Regression makes the up-and-down distances small and treats the two features differently. PCA makes the right-angle distances small and treats both features alike, so the two lines differ.
- "A large share means PC1 is good for predicting an outcome." PCA never sees an outcome. The direction with the most spread need not be the one that matters for a prediction.
- "The signs of PC1 mean something." A direction and its opposite give the same line. Software may flip the sign; the line stays put.
Further reading
- James, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning (2nd ed.), Section 12.2, "Principal Components Analysis". It covers the maximum-variance view (12.2.1), the closest-line view (12.2.2), the proportion of variance explained (12.2.3) and "Scaling the Variables" (12.2.4). Its variances divide by n, so its version of the step 2 identity has no n ÷ (n − 1) factor.
- Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Chapter 10, "Dimensionality Reduction with Principal Component Analysis", for the linear algebra in full.
About the data. The 50 people are made up. Height = 170 + 8 z₁ cm and arm span = 0.9 × (height − 170) + 170 + 4 z₂ cm, where z₁ and z₂ are independent draws from a normal distribution with mean 0 and SD 1, made with mulberry32, seed 111, drawn z₁ then z₂ for each person in turn. Step 6 uses the same people with arm span × 10, in mm. The ringed person is the first one drawn. Variances and covariances divide by n − 1 = 49 (sample), the mean squared error in step 5 divides by n = 50, and standardising uses the population SD (divide by n). Displayed values are rounded; the calculations are not.
If this helped, you can .