Skip to the lesson
ML Explorer Reju Sam John

ML Explorer · Level 2 · Learning from data

Many features and the loss surface

With two numbers to learn, the loss becomes a surface. Its shape decides how easily gradient descent finds the bottom.

About 12 minutes · No maths needed; the maths is there if you want it

Overview

Lesson 7 learned one number. Here the model learns two, the intercept and the slope of a line, and we draw the loss as a map with height lines.

The feature x runs from about 100 to 300, the size of a floor area in square metres, say. Numbers that size turn the map into a long, narrow valley where gradient descent struggles. Standardising x fixes that.

The points are made up. See "About the data" at the end.

You'll learn

  • how to read a contour map of a loss with two inputs
  • what the gradient is, and why gradient descent steps against it
  • why an unscaled feature slows gradient descent down, and how standardising fixes it
  1. Two numbers, one surface

    The model is the line ŷ = a + w·x. Every pair of values (a, w) has a loss, and all the pairs together make a surface. We draw it from above, as a contour map.

    a is the intercept, as in Lesson 6. w is the slope, called the weight, as in Lesson 7. This time we do not centre the data, so both numbers have to be learned.

    Each point on the map is one pair (a, w), so one line, and its loss is that line's mean squared error (MSE). The stage works the MSE out on a fine grid of pairs and joins up pairs with equal loss. Each ring (most run off the edge here) is one such height line. Inner rings are lower; the violet square marks the lowest point.

    Drag the dot, or use the sliders. The line on the right moves with it.

    The two axes are stretched by different amounts: one unit of w is far taller on screen than one unit of a. Without that, the rings would be too thin to see.

    The maths

    With nn points and residuals ei=yi−(a+w xi)e_i = y_i - (a + w\,x_i), the loss is

    L(a,w)=1n∑i=1nei2L(a, w) = \frac{1}{n}\sum_{i=1}^{n} e_i^2

    A contour, or level set, is every pair with the same loss: {(a,w):L(a,w)=c}\{(a, w) : L(a, w) = c\} for a fixed number cc. The map draws six of them.

  2. The gradient points uphill

    The gradient puts the slope of the loss in each direction together. It points uphill, so gradient descent steps the opposite way.

    Hold w fixed and nudge a: the loss changes at some rate, the slope in the a direction. Do the same for w. These two slopes are partial derivatives, and together they are the gradient, written ∇L. Lesson 5 gave a first look at it.

    Wherever the gradient is not zero, it points the way the loss rises fastest, uphill. The dashed arrow shows it. A step of gradient descent goes the opposite way, downhill, shown by the solid arrow.

    The read-out gives the two slopes. At most positions the slope for w is far bigger than the slope for a. Nudge w by some amount and each prediction moves x times as far as when you nudge a by the same amount, and here x is between 100 and 300.

    The arrows show direction only. Because the axes are stretched, they need not look square-on to the rings.

    The maths

    Differentiating the loss with respect to each number in turn:

    ∂L∂a=−2n∑i=1nei∂L∂w=−2n∑i=1nei xi\begin{aligned} \frac{\partial L}{\partial a} &= -\frac{2}{n}\sum_{i=1}^{n} e_i \\ \frac{\partial L}{\partial w} &= -\frac{2}{n}\sum_{i=1}^{n} e_i\,x_i \end{aligned}

    so, with θ=(a,w)\boldsymbol{\theta} = (a, w),

    ∇L(θ)=−2n∑i=1nei (1, xi)\nabla L(\boldsymbol{\theta}) = -\frac{2}{n}\sum_{i=1}^{n} e_i\,(1,\ x_i)

    A small move δ\boldsymbol{\delta} changes the loss by about ∇L⋅δ\nabla L \cdot \boldsymbol{\delta}. Among moves of a given length, that is largest when δ\boldsymbol{\delta} points along ∇L\nabla L, and most negative when it points the opposite way. Gradient descent takes

    θk+1=θk−η ∇L(θk)\boldsymbol{\theta}_{k+1} = \boldsymbol{\theta}_k - \eta\,\nabla L(\boldsymbol{\theta}_k)
  3. Raw x: a long, narrow valley

    With x in the hundreds, the rings are long and thin: a narrow valley, steep across and almost flat along. Gradient descent bounces across it and barely moves along it.

    The run starts from the line ŷ = 0, at a = 0, w = 0. Lesson 7 showed that too big an η makes the steps grow. Here the steepest direction sets that limit, and both runs in this lesson use the same rule: η is 90% of the largest stable value for its own bowl (step 5 explains the limit).

    The chart shows the extra loss (the loss minus the lowest possible loss) after each step, on a log scale. It falls fast while the bouncing dies down, then goes almost flat.

    Each step is a straight move; the dashed arcs only keep hops that retrace each other apart.

    The maths

    The second derivatives of the loss form the Hessian matrix:

    H=2n(n∑ixi∑ixi∑ixi2)H = \frac{2}{n}\begin{pmatrix} n & \sum_i x_i \\ \sum_i x_i & \sum_i x_i^2 \end{pmatrix}

    Write θ∗\boldsymbol{\theta}^* for the lowest point and dk=θk−θ∗\mathbf{d}_k = \boldsymbol{\theta}_k - \boldsymbol{\theta}^* for the distance from it after kk steps. The loss is exactly a quadratic, so

    L(θ)=L(θ∗)+12 d⊤H ddk+1=(I−ηH) dk\begin{aligned} L(\boldsymbol{\theta}) &= L(\boldsymbol{\theta}^*) + \tfrac{1}{2}\,\mathbf{d}^{\top} H\, \mathbf{d} \\ \mathbf{d}_{k+1} &= (I - \eta H)\,\mathbf{d}_k \end{aligned}

    H has a steepest direction and a gentlest one, at right angles (on screen the stretch hides the right angle), with curvatures γmax⁡\gamma_{\max} and γmin⁡\gamma_{\min} (its eigenvalues). Along each, every step multiplies the distance to the bottom by q=1−η γq = 1 - \eta\,\gamma. Both runs use η=1.8/γmax⁡\eta = 1.8/\gamma_{\max}, below the stable limit 2/γmax⁡2/\gamma_{\max}.

    If the start is c1c_1 along the steep direction and c2c_2 along the gentle one, the extra loss after kk steps is

    L(θk)−L(θ∗)=12γmax⁡ c12 q12k+12γmin⁡ c22 q22k\begin{aligned} &L(\boldsymbol{\theta}_k) - L(\boldsymbol{\theta}^*) \\ &\quad = \tfrac{1}{2}\gamma_{\max}\, c_1^2\, q_1^{2k} + \tfrac{1}{2}\gamma_{\min}\, c_2^2\, q_2^{2k} \end{aligned}

    The step count is the smallest kk that brings this below 10−610^{-6} times its value at k=0k = 0. Here the gentle term decides it.

  4. Standardise x: a round bowl

    Replace each x by its z-score. The valley becomes a round bowl, and the same rule for η reaches the target in far fewer steps.

    Standardising (Lesson 2, Lesson 3) subtracts the mean of x and divides by its standard deviation. We use the population SD (divide by n), so z has mean 0 and SD 1. The model is now ŷ = az + wz·z.

    This map uses the same scale on both axes, so round rings look round. Every direction is equally steep.

    Every hop lies on the line through the start and the bottom, so the hops are drawn straight here. "Raw x" and "Standardised x" switch the map; the chart shows both runs.

    The maths

    With μ\mu the mean of x and σ\sigma its population SD,

    zi=xi−μσ1n∑izi=0,1n∑izi2=1\begin{aligned} z_i &= \frac{x_i - \mu}{\sigma} \\ \frac{1}{n}\sum_i z_i &= 0, \quad \frac{1}{n}\sum_i z_i^2 = 1 \end{aligned}

    so the Hessian of the new loss is

    H=2n(n00n)=2IH = \frac{2}{n}\begin{pmatrix} n & 0 \\ 0 & n \end{pmatrix} = 2I

    Both curvatures are 2, so η=1.8/2=0.9\eta = 1.8/2 = 0.9 and q=−0.8q = -0.8 in every direction. The extra loss is multiplied by 0.82=0.640.8^2 = 0.64 each step, so the count is the smallest kk with 0.64k≤10−60.64^k \le 10^{-6}.

    To map back, write y^=az+wz x−μσ\hat{y} = a_z + w_z\,\frac{x - \mu}{\sigma} and collect terms:

    w=wzσ,a=az−wz μσw = \frac{w_z}{\sigma}, \qquad a = a_z - \frac{w_z\,\mu}{\sigma}

    Every pair (az,wz)(a_z, w_z) gives the same predictions as its (a,w)(a, w), so the lowest loss is the same and so is the best line.

  5. Why: the condition number

    The steepest direction limits how big a step can be. The gentlest direction decides how many steps you need. Their ratio, the condition number, says how hard the bowl is.

    The curvature in a direction is how fast the slope changes as you move that way; high curvature means a steep, tight wall. The arrows mark the steepest and gentlest directions through the bottom of the raw bowl.

    η has to suit the steepest direction, or the steps there grow without end. In the gentle direction that same η is tiny, so each step there covers only a small fraction of the distance left. That is the crawl.

    Why so large for raw x? Its values are big and far from 0, so a and w get tangled: raise w a little and a must drop a lot to keep the line near the points. Centring x untangles them; dividing by the SD makes the curvatures equal.

    The maths
    κ=γmax⁡γmin⁡\kappa = \frac{\gamma_{\max}}{\gamma_{\min}}

    Stability needs 0<η<2/γmax⁡0 < \eta < 2/\gamma_{\max}. Then along the gentle direction

    q2=1−η γmin⁡>1−2κq_2 = 1 - \eta\,\gamma_{\min} > 1 - \frac{2}{\kappa}

    so each step removes less than a fraction 2/κ2/\kappa of the distance left along it. With a large κ\kappa and η\eta a fixed fraction of its limit, the number of steps grows roughly in proportion to κ\kappa.

    Most books write these eigenvalues as λ\lambda. This explorer keeps λ\lambda for the regularisation strength (Lesson 10), so here they are γ\gamma. Some books also define the condition number from the data matrix instead of the Hessian; that version is the square root of this one.

  6. More weights, the same idea

    A model with many features has one weight per feature. You can no longer draw the surface, but everything in this lesson still applies.

    With p features the model is ŷ = a + w₁x₁ + … + wpxp, so the loss depends on p + 1 numbers. The gradient has one slope per number, and each step moves them all at once.

    For a linear model with squared loss, the surface is still a bowl with one bottom, as long as no feature can be worked out exactly as a weighted sum of the others (or is constant). Its condition number still sets how hard the bottom is to reach.

    That is why each feature is usually standardised with its own mean and SD before training. That fixes features on very different scales. It does not fix two features that carry nearly the same information: a narrow valley can remain.

    Neural networks have losses that are not bowls (Lesson 7, step 6). Scaling the inputs still usually helps, but the neat picture here no longer holds exactly.

    On the right: the standardised run, mapped back, lies on top of the least-squares line.

    The maths

    Put a column of 1s and the pp features into a matrix XX, and the intercept and weights into a vector θ\boldsymbol{\theta}. Then

    L(θ)=1n ∥y−Xθ∥2H=2n X⊤X\begin{aligned} L(\boldsymbol{\theta}) &= \frac{1}{n}\,\lVert \mathbf{y} - X\boldsymbol{\theta} \rVert^2 \\ H &= \frac{2}{n}\,X^{\top} X \end{aligned}

    H is the same at every point, so the loss is a quadratic bowl. It has a single lowest point when no column of XX is a combination of the others; then every curvature is positive. Standardising feature jj uses zij=(xij−μj)/σjz_{ij} = (x_{ij} - \mu_j)/\sigma_j.

Quick check

On raw x, why does gradient descent bounce from wall to wall across the valley?

A smaller η would bounce less, and below half the stable limit it would not bounce at all. But it would crawl along the valley even more slowly. The bounce comes from the steep walls.

Right. The steep walls dominate the gradient, so each step points mostly across the valley. With η near its limit it lands on the far wall, while the move along the valley is tiny.

This loss is a single bowl with one bottom. The bounces are back and forth across one valley.

Every step uses all the data and the same formula, so nothing jumps at random. The curvatures come from x alone, through H; the noise in y moves the bottom but not the shape of the valley.

What does standardising x change?

Mapped back to the original units, the standardised fit gives the same intercept and slope as least squares on raw x.

Right. The condition number drops to 1, and gradient descent needs far fewer steps.

Each standardised pair makes the same predictions as some raw pair, so the losses on offer are the same, and so is the lowest one.

Only x is rescaled. The values to predict stay as they are.

Lesson complete

  • With two numbers to learn, the loss is a surface. A contour map shows it from above, with rings of equal loss.
  • The gradient collects the slopes in each direction. It points uphill, and gradient descent steps against it.
  • An unscaled feature makes a long, narrow valley: η must suit the steep walls, so the run bounces across and crawls along. The condition number measures this.
  • Standardising makes the bowl round, with condition number 1, and leaves the fitted line unchanged.

Common misconceptions

  • "Standardising changes the model's answer." For this linear model, mapped back, it gives the same line. It only makes the answer easier to find.
  • "To go faster along the valley, raise the learning rate." Past the stable limit the steps across the steep walls grow without end.
  • "Gradient descent heads straight for the bottom." It heads downhill from where it is, which in a narrow valley is mostly across it.
Next lesson: Train, validate, test →

Further reading

  1. Goodfellow, Bengio & Courville, Deep Learning, Section 4.3.1, "Beyond the Gradient: Jacobian and Hessian Matrices", which explains the condition number of the Hessian and shows gradient descent wasting time on the walls of a long, narrow canyon.
  2. Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 7.1, "Optimization Using Gradient Descent", whose remarks describe poorly conditioned problems as long, thin valleys in which gradient descent zigzags.

About the data. The 30 points are made up, from a seeded random number generator, mulberry32 with seed 13. For each point in turn, x is drawn uniformly between 100 and 300, then y = 50 + 0.9x plus 15 times a draw from a normal distribution with mean 0 and SD 1. The points miss the line only because of that noise; real data also has measurement error and things the model leaves out. Both runs start at ŷ = 0 with η = 1.8 divided by their bowl's steepest curvature. Step counts use the target in step 3 and the formula there; the standardised count was also checked by running it. The rings sit 25, 100, 400, 1,600, 6,400 and 25,600 above the lowest MSE. Standardising uses the population SD. Displayed values are rounded; the calculations are not.

If this helped, you can .