Handle Rare Outcomes with Care
Only a few deliveries are more than an hour late.
Handle Rare Outcomes with Care explained visually
Only a few deliveries are more than an hour late. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Only a few deliveries are more than an hour late.
A practical question
Only one trip in a hundred arrives more than an hour late. A model that always says 'no severe delay' scores ninety-nine percent accuracy and finds none of the events the dispatcher cares about.
The useful idea
Count the severe delays first. Then measure how many the model catches, how many false warnings it raises, and what each kind of error costs the workflow. A percentage without a numerator can hide that the evaluation contains only two positive examples.
Work through the example
Review individual severe-delay records for reliable labels and any special causes. Split data so similar repeated incidents don't leak into both training and test. A threshold chosen on the final test set would make the reported result too optimistic.
Limits and next step
Next, we'll check a subtler source of optimism: fields that weren't actually known when the warning would have been sent.
Worked example
Report how many severe delays occur before celebrating a high overall accuracy.
Quick check: test your understanding
Try it yourself: In the situation described above, explain the method and one reason its output might be misleading.
Check the reasoning
Rare outcomes need careful metrics, enough positive examples, and realistic review costs. A model predicting the common outcome every time can appear accurate while missing every rare case.
What is the main idea behind handle rare outcomes with care?
Rare outcomes need careful metrics, enough positive examples, and realistic review costs.
How is it used in the example?
Report how many severe delays occur before celebrating a high overall accuracy.
What limitation should be checked?
A model predicting the common outcome every time can appear accurate while missing every rare case.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 23 transcript135 words · Video transcript · Matches the published narration and captionsOpen transcript
Only one trip in a hundred arrives more than an hour late. A model that always says 'no severe delay' scores ninety-nine percent accuracy and finds none of the events the dispatcher cares about.
Count the severe delays first. Then measure how many the model catches, how many false warnings it raises, and what each kind of error costs the workflow. A percentage without a numerator can hide that the evaluation contains only two positive examples.
Review individual severe-delay records for reliable labels and any special causes. Split data so similar repeated incidents don't leak into both training and test. A threshold chosen on the final test set would make the reported result too optimistic.
Next, we'll check a subtler source of optimism: fields that weren't actually known when the warning would have been sent.