Precision and Coverage of a Rule
A warning rule catches many late trips but flags too many on-time ones.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
A warning rule catches many late trips but flags too many on-time ones.
A practical question
Our rainy-long-route rule was correct on its only matching late trip. That is perfect precision on one warning, but it may be nearly useless if hundreds of other late trips go unflagged.
The useful idea
Coverage counts how many trips match the rule. Precision asks what fraction of matched warning trips were actually late. Also measure how many of all late trips the rule finds if that matters to the dispatcher. Each denominator answers a different question.
Work through the example
Report raw counts beside percentages on held-out weeks. Inspect false warnings and missed severe delays. A rule covering one trip is too fragile to celebrate based on its score alone. Precision answers how often a triggered rule was right; coverage answers how often the rule applies. Report counts alongside both rates. A rule that is perfect on two deliveries tells a different story from one that is accurate on hundreds. Test on a later period and inspect conditions where the rule fails. The operational choice depends on whether a narrow, high-confidence alert or a broader screening rule is more useful to the team. Use this case to test your understanding: Report both counts and rates on held-out deliveries. Here is the limitation to watch for: A precise rule covering only one case may be of little operational use. This supports a careful conclusion: Rule coverage measures how many examples match; precision measures how often matched cases are actually positive. Put the numerator and denominator next to each reported rule rate. One correct alert out of one trigger should not sound stronger than hundreds of reviewed trips. Also count the late deliveries outside the rule's coverage before deciding how useful it is. Keep the future period separate from rule tuning.
Limits and next step
Next, we'll combine several understandable rules and decide what happens when their instructions conflict.
Worked example
Report both counts and rates on held-out deliveries.
Quick check: test your understanding
Try it yourself: Consider this case: Report both counts and rates on held-out deliveries. Explain how you would check the result. In your answer, address this limitation: A precise rule covering only one case may be of little operational use.
Check the reasoning
Rule coverage measures how many examples match; precision measures how often matched cases are actually positive. A precise rule covering only one case may be of little operational use.
What is the main idea behind precision and coverage of a rule?
Rule coverage measures how many examples match; precision measures how often matched cases are actually positive.
How is it used in the example?
Report both counts and rates on held-out deliveries.
What limitation should be checked?
A precise rule covering only one case may be of little operational use.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 62 transcript297 words · Narration ready for the upcoming videoOpen transcript
Our rainy-long-route rule was correct on its only matching late trip. That is perfect precision on one warning, but it may be nearly useless if hundreds of other late trips go unflagged.
Coverage counts how many trips match the rule. Precision asks what fraction of matched warning trips were actually late. Also measure how many of all late trips the rule finds if that matters to the dispatcher. Each denominator answers a different question.
Report raw counts beside percentages on held-out weeks. Inspect false warnings and missed severe delays. A rule covering one trip is too fragile to celebrate based on its score alone. Precision answers how often a triggered rule was right; coverage answers how often the rule applies. Report counts alongside both rates. A rule that is perfect on two deliveries tells a different story from one that is accurate on hundreds. Test on a later period and inspect conditions where the rule fails. The operational choice depends on whether a narrow, high-confidence alert or a broader screening rule is more useful to the team. Use this case to test your understanding: Report both counts and rates on held-out deliveries. Here is the limitation to watch for: A precise rule covering only one case may be of little operational use. This supports a careful conclusion: Rule coverage measures how many examples match; precision measures how often matched cases are actually positive. Put the numerator and denominator next to each reported rule rate. One correct alert out of one trigger should not sound stronger than hundreds of reviewed trips. Also count the late deliveries outside the rule's coverage before deciding how useful it is. Keep the future period separate from rule tuning.
Next, we'll combine several understandable rules and decide what happens when their instructions conflict.