Gradients and Derivatives
Interpret derivatives as local rates of change and gradients as directions in a multi-parameter loss landscape.
Gradients and Derivatives explained visually
Use a mountain road to interpret positive, negative, and near-zero derivatives before forming a gradient.
What you will learn
Trace derivatives, gradients, and parameter responsibility through a neural network.
Interpret the local derivative
A derivative measures how quickly one output changes when one input changes near the current point.
Use sign and magnitude
The sign indicates direction, while magnitude describes local sensitivity.
Combine partial derivatives
A gradient collects the derivatives for many parameters into one vector.
Move against the gradient
Gradient descent moves opposite the direction of steepest local loss increase and recalculates after every update.
Quick check: test your understanding
What does a derivative near zero indicate?
The function is nearly flat at that point.
Which way does a gradient point?
Toward the steepest local increase.
Why is the gradient recalculated?
The local slope changes after the parameters move.
Accessible referenceLesson 34 transcript382 words · Matches the published narration and captionsOpen transcript
Backpropagation depends on gradients. To understand a gradient, begin with the simpler idea of a derivative. Imagine a cyclist traveling along a mountain road. At one point, the road rises steeply. At another point, it becomes nearly flat. The slope tells the cyclist how quickly the elevation changes as
the bicycle moves forward. A derivative plays the same role for a mathematical function. It measures how quickly an output changes when one input changes. If a small increase in the input produces a large increase in the output, the derivative has a large positive value.
If the output decreases, the derivative is negative. A value near zero means the function is almost flat at that point. During neural-network training, the function we care about is the loss. The inputs to that function include the model's weights and biases. We want to know how the loss responds when those parameters
change. Suppose one weight has a positive derivative with respect to the loss. Increasing that weight would increase the loss. To reduce the loss, the optimizer should move the weight in the opposite direction. If the derivative is negative, increasing the weight would reduce the
loss. The optimizer can therefore increase it. The magnitude also matters. A large magnitude suggests that a small parameter change can have a strong effect. A small magnitude suggests a weaker effect. A neural network contains many parameters, not just one. The gradient collects the derivatives for all relevant parameters into one
structure. You can imagine it as an arrow in a landscape with thousands or millions of dimensions. That arrow points toward the steepest increase in loss. Gradient descent moves in the opposite direction because training aims to decrease the loss. The learning rate determines how far the optimizer moves during each
update. The gradient is local information. It describes the slope near the network's current parameter values. After the parameters move, the slope may change, so the network calculates new gradients during the next training step. This explains why training requires repeated forward and backward passes.
Each pass provides updated information about the surrounding loss landscape. A derivative handles one changing input. Neural networks contain functions with many inputs, so we also need partial derivatives. We will connect them through the chain rule in Lesson 35.