Evaluate Errors Across Groups
One district receives many unnecessary delay alerts.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
One district receives many unnecessary delay alerts.
A practical question
One district receives many unnecessary delay warnings. The overall model score looks fine, but the dispatcher there spends much more time checking false alarms.
The useful idea
Compare false-positive and missed-delay rates across relevant districts, routes, and conditions. Show the number of trips in each group. A difference from ten examples is far less stable than one from ten thousand, but neither should be dismissed without inspection.
Work through the example
Review samples and ask whether data coverage, route availability, label policy, or a genuine model failure explains the pattern. Test fixes on held-out trips for all groups; reducing one district's false alarms should not quietly increase missed delays elsewhere. An overall error rate may hide a depot or neighborhood where warnings often fail. Report each group's sample count, false alarms, and missed late trips on a held-out period. Ask whether some routes were unavailable or dispatched differently before attributing a gap to the model. Small groups have noisy rates, so show uncertainty rather than ranking them confidently. A fairer evaluation starts by understanding what decision each group actually faced. Use this case to test your understanding: Report group counts and inspect whether route availability explains a difference. Here is the limitation to watch for: An aggregate accuracy score can hide concentrated harm. This supports a careful conclusion: Compare false-positive and false-negative rates across relevant groups and conditions. A district can show many false alerts because its trips are different, its labels are inconsistent, or the model behaves differently there. Compare these explanations using counts and examples before assigning a cause. A group rate alone cannot distinguish them. Inspect a few source records and the route policy for that district, then test whether the same disparity persists in a later period. Do not infer a cause from the chart alone.
Limits and next step
Next, we'll monitor the live system as schedules and trip patterns change over time.
Worked example
Report group counts and inspect whether route availability explains a difference.
Quick check: test your understanding
Try it yourself: Consider this case: Report group counts and inspect whether route availability explains a difference. Explain how you would check the result. In your answer, address this limitation: An aggregate accuracy score can hide concentrated harm.
Check the reasoning
Compare false-positive and false-negative rates across relevant groups and conditions. An aggregate accuracy score can hide concentrated harm.
What is the main idea behind evaluate errors across groups?
Compare false-positive and false-negative rates across relevant groups and conditions.
How is it used in the example?
Report group counts and inspect whether route availability explains a difference.
What limitation should be checked?
An aggregate accuracy score can hide concentrated harm.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 74 transcript307 words · Narration ready for the upcoming videoOpen transcript
One district receives many unnecessary delay warnings. The overall model score looks fine, but the dispatcher there spends much more time checking false alarms.
Compare false-positive and missed-delay rates across relevant districts, routes, and conditions. Show the number of trips in each group. A difference from ten examples is far less stable than one from ten thousand, but neither should be dismissed without inspection.
Review samples and ask whether data coverage, route availability, label policy, or a genuine model failure explains the pattern. Test fixes on held-out trips for all groups; reducing one district's false alarms should not quietly increase missed delays elsewhere. An overall error rate may hide a depot or neighborhood where warnings often fail. Report each group's sample count, false alarms, and missed late trips on a held-out period. Ask whether some routes were unavailable or dispatched differently before attributing a gap to the model. Small groups have noisy rates, so show uncertainty rather than ranking them confidently. A fairer evaluation starts by understanding what decision each group actually faced. Use this case to test your understanding: Report group counts and inspect whether route availability explains a difference. Here is the limitation to watch for: An aggregate accuracy score can hide concentrated harm. This supports a careful conclusion: Compare false-positive and false-negative rates across relevant groups and conditions. A district can show many false alerts because its trips are different, its labels are inconsistent, or the model behaves differently there. Compare these explanations using counts and examples before assigning a cause. A group rate alone cannot distinguish them. Inspect a few source records and the route policy for that district, then test whether the same disparity persists in a later period. Do not infer a cause from the chart alone.
Next, we'll monitor the live system as schedules and trip patterns change over time.