Fine-Tuning Without Forgetting Everything
The new classification head is working, but some target images still look different from the data used to pretrain the backbone.
Fine-Tuning Without Forgetting Everything explained visually
The new classification head is working, but some target images still look different from the data used to pretrain the backbone. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
The new classification head is working, but some target images still look different from the data used to pretrain the backbone.
Adapt gradually
The new classification head is working, but some target images still look different from the data used to pretrain the backbone. Fine-tuning allows selected pretrained layers to adapt. This can improve the representation for the new task, but careless updates can also destroy useful features, especially with a small or noisy dataset.
Control the update size
A practical starting point is to train the head, then unfreeze selected later layers with a smaller learning rate. Earlier layers may remain fixed while later representations adjust. This is a strategy to evaluate, not a universal rule. Batch normalization and other stateful components need deliberate handling because their running statistics may change even when some parameters are frozen, depending on the framework and training mode.
Keep the validated version
Monitor held-out performance and save successful checkpoints. Compare the frozen-backbone baseline with the fine-tuned result under the same split and metric. If fine-tuning makes training accuracy improve while validation accuracy falls, the additional flexibility may be fitting noise or limited coverage. Keep the previous validated model available. A later training run should replace it only after evaluation shows that the new artifact meets the chosen requirements.
Worked example
A frozen-backbone model achieves 82 percent validation accuracy, while a fine-tuned run reaches 97 percent training accuracy but only 76 percent validation accuracy. The higher training score is not evidence of a better final model. The frozen baseline remains stronger under this validation comparison.
Quick check: test your understanding
Try it yourself: Name two settings you would reconsider when fine-tuning quickly harms validation performance.
Check the reasoning
Consider a smaller learning rate, fewer unfrozen layers, stronger regularization, better data coverage, or earlier stopping. Evaluate changes using the same valid development split.
Does freezing parameter gradients necessarily freeze every form of model state?
No. Some layers maintain running statistics or other state that depends on training mode.
Further reading
Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.
Deep Learning — Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 72 transcript230 words · Video transcript · Matches the published narration and captionsOpen transcript
The new prediction head is trained, but the results could be better. Should we let the rest of the network change too? That next step is called fine-tuning.
With a frozen backbone, we only trained the new head. During fine-tuning, we allow some pretrained layers to update as well. Now the features themselves can become better suited to our images.
A sensible experiment is to unfreeze a few later layers and use a smaller learning rate for the backbone. We want to adjust useful features without immediately replacing everything the model learned before. This is a starting strategy, not a rule that wins on every dataset.
One detail deserves care. Freezing parameters does not necessarily freeze every piece of model state. Batch normalization, for example, can update its running statistics depending on the training mode. Check that behavior in your implementation.
Now look at the evidence. Our frozen model gets eighty-two percent validation accuracy. After fine-tuning, training accuracy rises to ninety-seven percent, but validation accuracy falls to seventy-six percent. Which model would you keep? Under this comparison, the frozen model still performs better on unseen data.
Try smaller updates, fewer unfrozen layers, or better training examples, then compare again using the same evaluation setup. Save your strongest checkpoint. Fine-tuning is useful when it improves the task you actually care about, not simply when it makes the training score look impressive.