Diagnosing High Variance
Recognize when strong training performance fails to transfer to new examples.
Diagnosing High Variance explained visually
Follow a memorizing student and a wolf classifier to diagnose high variance and unreliable shortcuts.
What you will learn
Read training signals and distinguish high bias from high variance.
Training memory is not generalization
A flexible model can memorize details or shortcuts in its training set instead of learning patterns that transfer to new examples.
Watch the validation curve
Training loss may continue decreasing while validation loss stops improving and begins to rise, producing a growing performance gap.
Recognize the risk factors
Small datasets, excessive capacity, duplicates, noisy labels, and narrow collection conditions can reinforce fragile patterns.
Improve transfer to new data
More varied data, realistic augmentation, regularization, a smaller model, and early stopping can reduce high variance when they address the observed cause.
Quick check: test your understanding
What is high variance commonly associated with?
Overfitting.
What curve pattern warns us?
Training improves while validation performance stops improving or worsens.
What does early stopping preserve?
The model from before validation performance begins to deteriorate.
Accessible referenceLesson 31 transcript387 words · Matches the published narration and captionsOpen transcript
High bias prevents a model from learning enough. High variance creates a different problem. Imagine a student who memorizes every answer from one practice exam, including the order of the choices. When the real exam presents different questions, that perfect memory no longer helps. A neural network can behave in the same way.
Consider a wildlife classifier trained to distinguish wolves from dogs. Every wolf photograph in the training set contains snow. Every dog photograph contains green grass. A flexible network may discover an easy shortcut. Instead of learning the animal's faces, ears, and body shapes, it associates snow with wolves
and grass with dogs. Training accuracy becomes extremely high. Now show the network a dog standing in snow. The prediction fails because the model memorized an accidental background pattern. This behavior represents high variance, commonly associated with overfitting. Its learning curves often reveal a clear warning.
Training loss continues to decrease because the model fits the training examples more closely. Validation loss decreases at first while the network learns useful patterns. Later, validation loss stops improving and begins to rise. The gap between training and validation performance becomes larger. Continuing the same training may make that gap worse.
Several conditions increase the risk. A small data set gives the model fewer examples of real variation. A model with excessive capacity can memorize tiny details. Duplicate images, noisy labels, and narrow camera conditions can reinforce unreliable shortcuts. Possible solutions target those causes. More varied training data can show wolves and dogs under different backgrounds, lighting
conditions, distances, and camera angles. Data augmentation can create realistic variations when it preserves the correct labels. Dropout and weight regularization can discourage fragile reliance on a narrow group of connections. A smaller model may also generalize better when the original network contains unnecessary capacity.
Early stopping provides another practical defense. We monitor validation performance and save the model from the point before validation loss begins to deteriorate. High training performance alone cannot prove that a model works. The model must succeed on separate examples that represent its real deployment conditions.
We have now identified the two major extremes. High bias means the model learns too little. High variance means it learns the training examples too specifically. In the next lesson, we will bring both ideas together and select a practical balance.