Bootstrap Samples and Bagging
The team trains several trees on different samples of deliveries.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The team trains several trees on different samples of deliveries.
A practical question
A small change in training deliveries produces a very different decision tree. Bagging trains several trees on different bootstrap samples and combines their predictions to reduce some of that instability.
The useful idea
Each bootstrap sample draws training records with replacement. Some trips appear more than once in a sample; others do not appear in that sample at all. The trees can therefore make different decisions. Voting or averaging can smooth their disagreement, though it cannot supply missing information about an unobserved route.
Work through the example
Compare one tree with a bagged collection on held-out weeks. Look at a borderline trip where individual trees disagree. If every tree shares the same blind spot, the combined vote may still be confidently wrong. Bagging trains each tree on a different sample of historical deliveries, drawn with replacement. Some trips appear several times in a sample and others not at all. Compare predictions from one tree with the vote or average across the collection on later trips. The ensemble can reduce instability, but several trees trained on the same biased history may share the same blind spot. Inspect errors during road closures rather than assuming that more trees guarantee a trustworthy answer. Now check the idea on a concrete example: Draw training records with replacement for each tree and compare voting with one tree. A possible trap is worth naming: Bagging does not correct a systematic error shared by all trees. The answer follows from that evidence: Bagging uses bootstrap samples and aggregates model outputs to reduce some instability. If every tree learns from the same collection of unreliable labels, averaging cannot repair that problem. Select a missed severe delay and inspect whether different trees disagree or share the same blind spot. That distinction suggests whether diversity or better data is needed.
Limits and next step
Next, we'll make the trees less alike by changing which features they can consider at a split.
Worked example
Draw training records with replacement for each tree and compare voting with one tree.
Quick check: test your understanding
Try it yourself: Consider this case: Draw training records with replacement for each tree and compare voting with one tree. Explain how you would check the result. In your answer, address this limitation: Bagging does not correct a systematic error shared by all trees.
Check the reasoning
Bagging uses bootstrap samples and aggregates model outputs to reduce some instability. Bagging does not correct a systematic error shared by all trees.
What is the main idea behind bootstrap samples and bagging?
Bagging uses bootstrap samples and aggregates model outputs to reduce some instability.
How is it used in the example?
Draw training records with replacement for each tree and compare voting with one tree.
What limitation should be checked?
Bagging does not correct a systematic error shared by all trees.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 45 transcript310 words · Narration ready for the upcoming videoOpen transcript
A small change in training deliveries produces a very different decision tree. Bagging trains several trees on different bootstrap samples and combines their predictions to reduce some of that instability.
Each bootstrap sample draws training records with replacement. Some trips appear more than once in a sample; others do not appear in that sample at all. The trees can therefore make different decisions. Voting or averaging can smooth their disagreement, though it cannot supply missing information about an unobserved route.
Compare one tree with a bagged collection on held-out weeks. Look at a borderline trip where individual trees disagree. If every tree shares the same blind spot, the combined vote may still be confidently wrong. Bagging trains each tree on a different sample of historical deliveries, drawn with replacement. Some trips appear several times in a sample and others not at all. Compare predictions from one tree with the vote or average across the collection on later trips. The ensemble can reduce instability, but several trees trained on the same biased history may share the same blind spot. Inspect errors during road closures rather than assuming that more trees guarantee a trustworthy answer. Now check the idea on a concrete example: Draw training records with replacement for each tree and compare voting with one tree. A possible trap is worth naming: Bagging does not correct a systematic error shared by all trees. The answer follows from that evidence: Bagging uses bootstrap samples and aggregates model outputs to reduce some instability. If every tree learns from the same collection of unreliable labels, averaging cannot repair that problem. Select a missed severe delay and inspect whether different trees disagree or share the same blind spot. That distinction suggests whether diversity or better data is needed.
Next, we'll make the trees less alike by changing which features they can consider at a split.