Boosting: Improve a Model in Stages
A first model makes systematic mistakes.
Boosting: Improve a Model in Stages explained visually
A first model makes systematic mistakes. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A first model makes systematic mistakes.
Add corrective models
A first model makes systematic mistakes. Instead of replacing it completely, boosting builds an ensemble in stages, adding models that help improve the current predictions. Gradient boosting chooses these additions using information from the loss function. The idea is sequential correction, not simply training many independent models and voting.
Limit each update
Small decision trees are common building blocks. Each new tree contributes an adjustment, scaled by a learning rate. Smaller steps may require more trees but can improve control over fitting. Tree depth, number of rounds, sampling choices, and regularization affect the result. These settings interact, so tuning one number in isolation can be misleading.
Watch validation performance
Track validation performance as the ensemble grows. Training loss can keep improving after generalization stops improving. Early stopping can select a useful number of rounds using a properly separated validation set. Keep preprocessing and model selection inside the evaluation design. Boosting is powerful on many structured-data problems, but it is not automatically superior to every simpler model, especially when data are scarce or the task has a strong simple structure.
A worked example
Suppose an initial model predicts a delay of ten minutes. A new stage proposes a correction of four minutes. With a learning rate of one quarter, the update contributes one minute, so the ensemble predicts eleven minutes. The small step is intentional rather than a calculation error.
Worked example
Suppose an initial model predicts a delay of ten minutes. A new stage proposes a correction of four minutes. With a learning rate of one quarter, the update contributes one minute, so the ensemble predicts eleven minutes. The small step is intentional rather than a calculation error.
Quick check: test your understanding
Try it yourself: What should you do if training loss improves for many rounds while validation loss steadily worsens?
Check the reasoning
Investigate overfitting, use an earlier validated stopping point, and consider reducing complexity or strengthening regularization.
Are boosting stages normally independent of the earlier ensemble?
No. Each stage is trained in relation to the current model and objective.
How would you solve the practice case?
Investigate overfitting, use an earlier validated stopping point, and consider reducing complexity or strengthening regularization.
What does the worked example show?
Suppose an initial model predicts a delay of ten minutes. A new stage proposes a correction of four minutes. With a learning rate of one quarter, the update contributes one minute, so the ensemble predicts eleven minutes. The small step is intentional rather than a calculation error.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 10 transcript220 words · Video transcript · Matches the published narration and captionsOpen transcript
A first model makes systematic mistakes. Instead of replacing it completely, boosting builds an ensemble in stages, adding models that help improve the current predictions. Gradient boosting chooses these additions using information from the loss function. The idea is sequential correction, not simply training many independent models and voting.
Small decision trees are common building blocks. Each new tree contributes an adjustment, scaled by a learning rate. Smaller steps may require more trees but can improve control over fitting. Tree depth, number of rounds, sampling choices, and regularization affect the result. These settings interact, so tuning one number in isolation can be misleading.
Track validation performance as the ensemble grows. Training loss can keep improving after generalization stops improving. Early stopping can select a useful number of rounds using a properly separated validation set. Keep preprocessing and model selection inside the evaluation design. Boosting is powerful on many structured-data problems, but it is not automatically superior to every simpler model, especially when data are scarce or the task has a strong simple structure.
Suppose an initial model predicts a delay of ten minutes. A new stage proposes a correction of four minutes. With a learning rate of one quarter, the update contributes one minute, so the ensemble predicts eleven minutes. The small step is intentional rather than a calculation error.