ML Explorer · Level 2 · Learning from data
Many features and the loss surface
With two numbers to learn, the loss becomes a surface. Its shape decides how easily gradient descent finds the bottom.
Overview
Lesson 7 learned one number. Here the model learns two, the intercept and the slope of a line, and we draw the loss as a map with height lines.
The feature x runs from about 100 to 300, the size of a floor area in square metres, say. Numbers that size turn the map into a long, narrow valley where gradient descent struggles. Standardising x fixes that.
The points are made up. See "About the data" at the end.
You'll learn
- how to read a contour map of a loss with two inputs
- what the gradient is, and why gradient descent steps against it
- why an unscaled feature slows gradient descent down, and how standardising fixes it
-
Two numbers, one surface
The model is the line ŷ = a + w·x. Every pair of values (a, w) has a loss, and all the pairs together make a surface. We draw it from above, as a contour map.
a is the intercept, as in Lesson 6. w is the slope, called the weight, as in Lesson 7. This time we do not centre the data, so both numbers have to be learned.
Each point on the map is one pair (a, w), so one line, and its loss is that line's mean squared error (MSE). The stage works the MSE out on a fine grid of pairs and joins up pairs with equal loss. Each ring (most run off the edge here) is one such height line. Inner rings are lower; the violet square marks the lowest point.
Drag the dot, or use the sliders. The line on the right moves with it.
The two axes are stretched by different amounts: one unit of w is far taller on screen than one unit of a. Without that, the rings would be too thin to see.
The maths
With points and residuals , the loss is
A contour, or level set, is every pair with the same loss: for a fixed number . The map draws six of them.
The lowest point is Lesson 6's least-squares line for these points, at = and = , with MSE .
-
The gradient points uphill
The gradient puts the slope of the loss in each direction together. It points uphill, so gradient descent steps the opposite way.
Hold w fixed and nudge a: the loss changes at some rate, the slope in the a direction. Do the same for w. These two slopes are partial derivatives, and together they are the gradient, written ∇L. Lesson 5 gave a first look at it.
Wherever the gradient is not zero, it points the way the loss rises fastest, uphill. The dashed arrow shows it. A step of gradient descent goes the opposite way, downhill, shown by the solid arrow.
The read-out gives the two slopes. At most positions the slope for w is far bigger than the slope for a. Nudge w by some amount and each prediction moves x times as far as when you nudge a by the same amount, and here x is between 100 and 300.
The arrows show direction only. Because the axes are stretched, they need not look square-on to the rings.
The maths
Differentiating the loss with respect to each number in turn:
so, with ,
A small move changes the loss by about . Among moves of a given length, that is largest when points along , and most negative when it points the opposite way. Gradient descent takes
-
Raw x: a long, narrow valley
With x in the hundreds, the rings are long and thin: a narrow valley, steep across and almost flat along. Gradient descent bounces across it and barely moves along it.
The run starts from the line ŷ = 0, at a = 0, w = 0. Lesson 7 showed that too big an η makes the steps grow. Here the steepest direction sets that limit, and both runs in this lesson use the same rule: η is 90% of the largest stable value for its own bowl (step 5 explains the limit).
That gives η = . Each step jumps past the valley floor to the far wall, then back, a little less each time, while creeping along the valley. After steps, a has moved from 0 to , but the bottom is at a = .
The chart shows the extra loss (the loss minus the lowest possible loss) after each step, on a log scale. It falls fast while the bouncing dies down, then goes almost flat.
Call the run finished when the extra loss is below one millionth of its value at the start (the dotted target line, ). From the bowl's formula we can count the steps without running them: steps.
Each step is a straight move; the dashed arcs only keep hops that retrace each other apart.
The maths
The second derivatives of the loss form the Hessian matrix:
Write for the lowest point and for the distance from it after steps. The loss is exactly a quadratic, so
H has a steepest direction and a gentlest one, at right angles (on screen the stretch hides the right angle), with curvatures and (its eigenvalues). Along each, every step multiplies the distance to the bottom by . Both runs use , below the stable limit .
Here is across the valley (overshoot, shrinking) and along it (barely shrinking).
If the start is along the steep direction and along the gentle one, the extra loss after steps is
The step count is the smallest that brings this below times its value at . Here the gentle term decides it.
-
Standardise x: a round bowl
Replace each x by its z-score. The valley becomes a round bowl, and the same rule for η reaches the target in far fewer steps.
Standardising (Lesson 2, Lesson 3) subtracts the mean of x and divides by its standard deviation. We use the population SD (divide by n), so z has mean 0 and SD 1. The model is now ŷ = az + wz·z.
This map uses the same scale on both axes, so round rings look round. Every direction is equally steep.
The rule gives η = . Each step heads straight for the bottom and overshoots it by 0.8 of the distance that was left, so the bouncing dies away fast. The run reaches the same target in steps, against on raw x.
Every hop lies on the line through the start and the bottom, so the hops are drawn straight here. "Raw x" and "Standardised x" switch the map; the chart shows both runs.
Standardising changes the numbers, not the line. Turned back into the original units, this run gives a = and w = : Lesson 6's least-squares line.
The maths
With the mean of x and its population SD,
so the Hessian of the new loss is
Both curvatures are 2, so and in every direction. The extra loss is multiplied by each step, so the count is the smallest with .
To map back, write and collect terms:
Every pair gives the same predictions as its , so the lowest loss is the same and so is the best line.
Here = and = . The bottom of the round bowl is at = , = .
-
Why: the condition number
The steepest direction limits how big a step can be. The gentlest direction decides how many steps you need. Their ratio, the condition number, says how hard the bowl is.
The curvature in a direction is how fast the slope changes as you move that way; high curvature means a steep, tight wall. The arrows mark the steepest and gentlest directions through the bottom of the raw bowl.
η has to suit the steepest direction, or the steps there grow without end. In the gentle direction that same η is tiny, so each step there covers only a small fraction of the distance left. That is the crawl.
The condition number κ (kappa) is the steepest curvature divided by the gentlest. For raw x it is ( against ). For standardised x both curvatures are , so κ = , the best possible.
Why so large for raw x? Its values are big and far from 0, so a and w get tangled: raise w a little and a must drop a lot to keep the line near the points. Centring x untangles them; dividing by the SD makes the curvatures equal.
The maths
Stability needs . Then along the gentle direction
so each step removes less than a fraction of the distance left along it. With a large and a fixed fraction of its limit, the number of steps grows roughly in proportion to .
Most books write these eigenvalues as . This explorer keeps for the regularisation strength (Lesson 10), so here they are . Some books also define the condition number from the data matrix instead of the Hessian; that version is the square root of this one.
-
More weights, the same idea
A model with many features has one weight per feature. You can no longer draw the surface, but everything in this lesson still applies.
With p features the model is ŷ = a + w₁x₁ + … + wpxp, so the loss depends on p + 1 numbers. The gradient has one slope per number, and each step moves them all at once.
For a linear model with squared loss, the surface is still a bowl with one bottom, as long as no feature can be worked out exactly as a weighted sum of the others (or is constant). Its condition number still sets how hard the bottom is to reach.
That is why each feature is usually standardised with its own mean and SD before training. That fixes features on very different scales. It does not fix two features that carry nearly the same information: a narrow valley can remain.
Neural networks have losses that are not bowls (Lesson 7, step 6). Scaling the inputs still usually helps, but the neat picture here no longer holds exactly.
On the right: the standardised run, mapped back, lies on top of the least-squares line.
The maths
Put a column of 1s and the features into a matrix , and the intercept and weights into a vector . Then
H is the same at every point, so the loss is a quadratic bowl. It has a single lowest point when no column of is a combination of the others; then every curvature is positive. Standardising feature uses .
Quick check
Lesson complete
- With two numbers to learn, the loss is a surface. A contour map shows it from above, with rings of equal loss.
- The gradient collects the slopes in each direction. It points uphill, and gradient descent steps against it.
- An unscaled feature makes a long, narrow valley: η must suit the steep walls, so the run bounces across and crawls along. The condition number measures this.
- Standardising makes the bowl round, with condition number 1, and leaves the fitted line unchanged.
Common misconceptions
- "Standardising changes the model's answer." For this linear model, mapped back, it gives the same line. It only makes the answer easier to find.
- "To go faster along the valley, raise the learning rate." Past the stable limit the steps across the steep walls grow without end.
- "Gradient descent heads straight for the bottom." It heads downhill from where it is, which in a narrow valley is mostly across it.
Further reading
- Goodfellow, Bengio & Courville, Deep Learning, Section 4.3.1, "Beyond the Gradient: Jacobian and Hessian Matrices", which explains the condition number of the Hessian and shows gradient descent wasting time on the walls of a long, narrow canyon.
- Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 7.1, "Optimization Using Gradient Descent", whose remarks describe poorly conditioned problems as long, thin valleys in which gradient descent zigzags.
About the data. The 30 points are made up, from a seeded random number generator, mulberry32 with seed 13. For each point in turn, x is drawn uniformly between 100 and 300, then y = 50 + 0.9x plus 15 times a draw from a normal distribution with mean 0 and SD 1. The points miss the line only because of that noise; real data also has measurement error and things the model leaves out. Both runs start at ŷ = 0 with η = 1.8 divided by their bowl's steepest curvature. Step counts use the target in step 3 and the formula there; the standardised count was also checked by running it. The rings sit 25, 100, 400, 1,600, 6,400 and 25,600 above the lowest MSE. Standardising uses the population SD. Displayed values are rounded; the calculations are not.
If this helped, you can .