Vanishing, Exploding, and Dying Gradients
Recognize unstable gradient flow and select targeted methods that keep deep networks trainable.
Vanishing, Exploding, and Dying Gradients explained visually
Compare vanishing, exploding, and dying-ReLU failure modes and the methods used to control them.
What you will learn
Initialize, normalize, and stabilize signals and gradients in deeper networks.
Recognize vanishing gradients
Repeated multiplication by small derivatives can leave early layers with almost no useful update.
Identify dying ReLU
A ReLU neuron may remain inactive when its input and gradient stay on the zero side.
Control exploding gradients
Large repeated multipliers can create huge optimizer steps; norm clipping limits their length.
Monitor and stabilize training
Initialization, normalization, skip connections, gradient norms, and activation distributions help reveal and reduce instability.
Quick check: test your understanding
What happens when gradients vanish?
Early layers barely update.
How does leaky ReLU help?
It preserves a small slope for negative inputs.
What does gradient clipping mainly control?
Exploding gradients, not gradients that have already vanished.
Accessible referenceLesson 40 transcript385 words · Matches the published narration and captionsOpen transcript
Backpropagation sends gradients through every layer. In a deep network, that journey involves many multiplications. The values can become extremely small or extremely large. Imagine passing a correction through a long chain of amplifiers. If every amplifier multiplies the signal by zero point five, the
message becomes weaker at every stage. After many stages, almost nothing remains. This is the vanishing-gradient problem. Early layers receive gradients so small that their weights barely change. Training becomes slow, and the network struggles to learn features that depend on long chains of calculations.
Sigmoid and tanh can contribute to this problem when their inputs reach saturated regions. Their derivatives become small, and multiplying many small derivatives makes the backward signal shrink further. ReLU helps because its derivative remains one for positive inputs. Leaky ReLU and Parametric ReLU also keep a small slope for
negative inputs. However, ordinary ReLU creates another risk. If a neuron's input stays negative, its output and gradient remain zero. The neuron may stop updating and never become active again. This is called the dying-ReLU problem. Leaky ReLU reduces that risk by preserving a small negative slope
instead of returning a completely flat zero region. The opposite failure is an exploding gradient. If the backward path repeatedly multiplies values larger than one, the gradient can grow rapidly. The optimizer may take enormous steps, the loss may jump, and numerical values may overflow.
Good initialization helps control both problems. Xavier or He initialization keeps starting signals at a more useful scale. Batch normalization can stabilize internal activations. Skip connections create shorter paths through which information and gradients can travel. Gradient clipping directly addresses excessively large gradients. Clipping by value limits individual components.
Norm clipping measures the length of the complete gradient vector and scales it down when that length exceeds a threshold. The direction remains similar while the update becomes manageable. Gradient clipping primarily controls exploding gradients. It does not restore a gradient that has already vanished.
During training, we should monitor the loss, gradient norms, activation distributions, and the percentage of inactive ReLU neurons. These signals reveal instability before final accuracy hides the cause. Stable gradients allow deeper networks to learn useful representations. In Lesson 41, we will use those ideas to begin convolutional neural networks, models designed specifically for the spatial structure of images.