Uncertainty in an Estimated Score
A test accuracy changes when only a few deliveries are added.
Uncertainty in an Estimated Score explained visually
A test accuracy changes when only a few deliveries are added. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A test accuracy changes when only a few deliveries are added.
A practical question
Our validation score changes noticeably when five new late deliveries are added. Is the model unstable, or is the estimate noisy because the sample is small? Both are possible.
The useful idea
An interval around a measured score can describe sampling uncertainty under a specified method. It doesn't automatically account for a new rural depot, changed weather, or incorrect labels. Repeated valid group or time splits can show how sensitive results are to the evaluation sample.
Work through the example
Report the number of positive and negative trips along with the interval. Inspect a rare subgroup before claiming precise performance there. A narrow interval built from near-duplicate frames or trips would still answer the wrong population question.
Limits and next step
Next, we'll compare two candidate models on the same untouched test period without repeatedly choosing the winner from it.
Worked example
Compare score intervals from repeated valid splits or a suitable resampling method.
Quick check: test your understanding
Try it yourself: In the situation described above, explain the method and one reason its output might be misleading.
Check the reasoning
Estimated scores vary with sample size and sampling; intervals express uncertainty under assumptions. An interval does not repair a biased or leaky test set.
What is the main idea behind uncertainty in an estimated score?
Estimated scores vary with sample size and sampling; intervals express uncertainty under assumptions.
How is it used in the example?
Compare score intervals from repeated valid splits or a suitable resampling method.
What limitation should be checked?
An interval does not repair a biased or leaky test set.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 35 transcript129 words · Video transcript · Matches the published narration and captionsOpen transcript
Our validation score changes noticeably when five new late deliveries are added. Is the model unstable, or is the estimate noisy because the sample is small? Both are possible.
An interval around a measured score can describe sampling uncertainty under a specified method. It doesn't automatically account for a new rural depot, changed weather, or incorrect labels. Repeated valid group or time splits can show how sensitive results are to the evaluation sample.
Report the number of positive and negative trips along with the interval. Inspect a rare subgroup before claiming precise performance there. A narrow interval built from near-duplicate frames or trips would still answer the wrong population question.
Next, we'll compare two candidate models on the same untouched test period without repeatedly choosing the winner from it.