ML Explorer · Level 1 · Data & intuition
Slopes and derivatives
A slope says how steep a curve is at one point, and which way is downhill.
Overview
Training a model means making its loss as small as possible. Lesson 4 said the MSE curve is smooth, with a slope at every point. Lesson 7 follows that slope downhill. This lesson is the step in between.
It starts with the slope of a straight line, then moves to a curve, whose slope changes from place to place. The slope of a curve at a single point is called its derivative.
There is no data in this lesson, just one curve. See "About the curve" at the end.
You'll learn
- what a slope is, as rise over run
- how the slope at a single point comes from shrinking a gap towards zero
- why the sign of the slope tells you which way is downhill, and why the slope is 0 at the bottom
-
The slope of a straight line
The slope of a line says how steeply it climbs: how far it goes up for each step to the right. It is rise over run.
Pick two points on the line. The run is how far you move right to get from one to the other. The rise is how far you go up.
The slope is the rise divided by the run. The box works it out for the big triangle.
On a straight line, any two points give the same slope. The small triangle shows this: a smaller rise over a smaller run, but the same answer.
A positive slope climbs as you go right. A negative slope falls. A slope of 0 is flat.
The chart on the right shows the slope at each x. For a straight line it is the same everywhere, so the chart is flat.
The maths
Write the line as . Take two points on it, and , with . Then
The two points drop out, which is why every pair gives the same slope. Here the subscripts 1 and 2 just name the two points. This line is , so .
-
A curve's slope changes
A curve has no single slope. It is steep in some places and flatter in others.
The line has bent into a curve. We write f(x), said "f of x", for the height of the curve at x.
Zoom in close to any point on a smooth curve and it looks almost straight. The slope of that almost straight piece is the slope of the curve at that point.
The short line through the point is the tangent. It touches the curve there and heads the same way the curve does. Its slope is the curve's slope at that point. Step 3 shows how to work it out.
Move x with the slider. On the left the curve falls, so its slope is negative. On the right it climbs, so its slope is positive.
The chart on the right shows the slope at every x. It is no longer flat: the slope changes as x changes.
This is what Lesson 4 meant when it said the MSE curve "has a slope everywhere". It is smooth, so it has one at every point. The MAE curve has corners, and at a corner there is no single slope.
The maths
The curve is
drawn for . Its slope at is written , said "f prime of x", and called the derivative of at .
"Touches the curve and heads the same way" is a description, not a definition. Step 3 gives the exact definition. The tangent at is then the line through with slope .
-
Closing in on the slope
To find the slope at one point, take a second point a little way along and work out rise over run between them. Then bring the second point closer and closer.
The first point is at x. The second is at x + h, where h is a small step to the right.
The straight line through the two points is called a secant. Its slope is rise over run: the rise is f(x + h) − f(x), and the run is h.
Shrink h with the slider or the button. The secant swings round and closes in on the tangent, the dashed line.
The chart on the right shows the secant's slope for each h. As h shrinks, it heads for the tangent's slope. That value it heads for is the derivative, f′(x): the slope of the curve at x.
You cannot simply set h to 0. The run would be 0, and you cannot divide by 0. So the slider runs from 2 down to 0.01 and never reaches 0. The derivative is the value the secant's slope approaches, not its value at h = 0.
Here x stays at 3. The box shows the gap between the two slopes: halve h, and the gap halves too.
The maths
The derivative is defined as a limit:
"The limit as goes to 0" means the value the fraction gets as close to as you like by taking small enough, without ever setting . It is also written .
For this curve, expand and subtract :
Divide by :
So the secant's slope is the derivative plus exactly . The gap shrinks in proportion to , and as the slope goes to . The slider keeps positive. The definition also lets approach 0 through negative values, and for this curve that gives the same limit.
In practice nobody takes limits by hand each time. A few rules do the work. The power rule says the derivative of is . A number multiplying a term stays in front, the derivative of a sum is the sum of the derivatives, and a constant on its own has derivative 0. So
-
The sign tells you which way is downhill
A negative slope means the curve falls to the right, so downhill is to the right. A positive slope means downhill is to the left.
Move x and watch the arrow. It always points down the curve. Very close to the bottom there is no room to draw it, but the box still gives the direction.
Left of the bottom, the slope is negative, and going right takes you down. Right of the bottom, the slope is positive, and going left takes you down.
So to go downhill, move x the opposite way to the slope's sign.
The size of the slope matters too. The steeper the curve, the further its slope is from 0, and the more the height changes for a small step.
The rule holds for small steps. Take too big a step and you can overshoot the bottom and end up higher on the other side. Lesson 7 builds gradient descent on this rule, and shows what the step size does.
The maths
For this curve, the same expansion as in step 3 gives, exactly,
For a small step , the term is tiny, so the change in height is close to . When , a step to the right () makes that negative, and the curve goes down. When , a step to the left () does.
How small is small? Here exactly when . For a step to the right from a point with , that means . A longer step overshoots and lands at least as high as where it started.
-
At the bottom the slope is 0
At the bottom of the bowl the curve is flat for an instant, so its slope is 0.
The stage has moved x to the lowest point. The tangent is flat, and the slope chart crosses 0 at the same x.
So one way to find the bottom is to look for where the slope is 0. On this curve that happens at just one x.
A slope of 0 does not always mean a minimum. It can also be a peak, or a spot where the curve levels off and then carries on the same way. For this bowl-shaped curve, it is the minimum.
Move x away and the slope is no longer 0.
A look ahead. A function can have more than one input. Then you can take its slope along one input at a time, holding the others fixed. That is a partial derivative. Put the partial derivatives together, one per input, and you have the gradient. Wherever it is not zero, it points uphill, the way the function rises fastest. Lesson 8 uses it on a loss with two inputs.
The maths
Set the slope to 0 and solve:
The slope of the slope, , is positive everywhere. The slope keeps rising as grows, from negative to positive, which is what makes the curve a bowl and its flat point the minimum. For contrast, has slope 0 at its peak, , and has slope 0 at , where it is neither a peak nor a minimum.
For a function of two inputs, say , the partial derivative is the slope as changes with held fixed, and the other way round. The gradient is the vector of both:
For example, has and . Wherever the gradient is not zero, it points in the direction in which rises fastest.
Quick check
Lesson complete
- The slope of a straight line is rise over run, and it is the same everywhere on the line.
- A curve's slope changes from point to point. The slope at one point, the derivative, is the limit of a secant's slope as the gap h shrinks to 0.
- A negative slope means downhill is to the right, a positive slope means it is to the left, and at the bottom of a bowl the slope is 0.
Common misconceptions
- "The derivative is the secant's slope with h = 0." At h = 0 the run is 0, and you cannot divide by 0. The derivative is the value the secant's slope approaches as h shrinks.
- "A bigger slope means a higher point." The slope measures how steep the curve is, not how high. At the lowest point here the slope is 0.
- "A slope of 0 always means a minimum." It can also be a peak, or a point where the curve levels off and carries on.
- "Going against the slope always lowers the curve." Only for small enough steps. A big step can overshoot the bottom.
Further reading
- Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Section 5.1, "Differentiation of Univariate Functions", which builds the derivative from the difference quotient (the secant's slope) and works through a polynomial example, and Section 5.2, "Partial Differentiation and Gradients".
- Goodfellow, Bengio & Courville, Deep Learning, Section 4.3, "Gradient-Based Optimization", which uses the sign of the derivative to move downhill, and explains why a slope of 0 can be a minimum, a maximum or neither.
About the curve. This lesson uses no data, random or otherwise, so there is no seed. The curve is f(x) = 0.5x² − 2x + 3, drawn for x from −1 to 6. It was chosen because its numbers come out round: its slope is f′(x) = x − 2, and its lowest point is at x = 2, where f(x) = 1. The straight line in step 1 is y = 0.5 + 1.5x. In step 3, x stays at 3 and h runs from 0.01 to 2, so it never reaches 0. The stage shows the slope of the straight line in step 1 as it comes out, values to two decimal places in steps 2, 4 and 5, and to three in step 3; the calculations use unrounded values.
If this helped, you can .