Choose a Classification Threshold
The dispatcher can review only a limited number of warnings.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The dispatcher can review only a limited number of warnings.
A practical question
The dispatcher can review thirty warnings a day. Our model produces a risk score for each trip, but it cannot choose the warning policy by itself. We need a threshold that fits the cost of missed delays and the team's review capacity.
The useful idea
Lower the threshold and more trips are flagged. Some additional late deliveries will be caught, along with more false alarms. Raise it and the queue shrinks, but more late trips may pass unnoticed. The exact trade-off depends on the score distribution in the real deployment population.
Work through the example
On validation trips, show a confusion matrix and estimated warning count for several choices. Pick a threshold before the final test, then report how it performed there. If conditions change, revisit the policy with fresh evidence. The dispatcher can only inspect a finite number of warnings each day. Lay out several possible score thresholds and count how many trips each would flag on a held-out validation period. Put the missed late deliveries beside the false alarms, then discuss their different costs. A threshold that looks good on one season may overload the team during holiday traffic. Decide the policy before opening the final test set, and monitor workload as well as classification accuracy when the system runs. Use this case to test your understanding: Choose a threshold using validation costs and the team's review capacity. Here is the limitation to watch for: Do not pick the threshold by repeatedly inspecting the final test period. This supports a careful conclusion: A classification threshold converts a score into an action and changes false-alarm versus missed-delay rates. Take a day with more trips than the review team can handle. Count alerts under the chosen threshold and decide which kinds of missed delay are most costly. A threshold is an operational policy, so its workload and mistakes both belong in the report.
Limits and next step
Next, we'll predict from similar past trips and examine what the word 'similar' means mathematically.
Worked example
Choose a threshold using validation costs and the team's review capacity.
Quick check: test your understanding
Try it yourself: Consider this case: Choose a threshold using validation costs and the team's review capacity. Explain how you would check the result. In your answer, address this limitation: Do not pick the threshold by repeatedly inspecting the final test period.
Check the reasoning
A classification threshold converts a score into an action and changes false-alarm versus missed-delay rates. Do not pick the threshold by repeatedly inspecting the final test period.
What is the main idea behind choose a classification threshold?
A classification threshold converts a score into an action and changes false-alarm versus missed-delay rates.
How is it used in the example?
Choose a threshold using validation costs and the team's review capacity.
What limitation should be checked?
Do not pick the threshold by repeatedly inspecting the final test period.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 40 transcript321 words · Narration ready for the upcoming videoOpen transcript
The dispatcher can review thirty warnings a day. Our model produces a risk score for each trip, but it cannot choose the warning policy by itself. We need a threshold that fits the cost of missed delays and the team's review capacity.
Lower the threshold and more trips are flagged. Some additional late deliveries will be caught, along with more false alarms. Raise it and the queue shrinks, but more late trips may pass unnoticed. The exact trade-off depends on the score distribution in the real deployment population.
On validation trips, show a confusion matrix and estimated warning count for several choices. Pick a threshold before the final test, then report how it performed there. If conditions change, revisit the policy with fresh evidence. The dispatcher can only inspect a finite number of warnings each day. Lay out several possible score thresholds and count how many trips each would flag on a held-out validation period. Put the missed late deliveries beside the false alarms, then discuss their different costs. A threshold that looks good on one season may overload the team during holiday traffic. Decide the policy before opening the final test set, and monitor workload as well as classification accuracy when the system runs. Use this case to test your understanding: Choose a threshold using validation costs and the team's review capacity. Here is the limitation to watch for: Do not pick the threshold by repeatedly inspecting the final test period. This supports a careful conclusion: A classification threshold converts a score into an action and changes false-alarm versus missed-delay rates. Take a day with more trips than the review team can handle. Count alerts under the chosen threshold and decide which kinds of missed delay are most costly. A threshold is an operational policy, so its workload and mistakes both belong in the report.
Next, we'll predict from similar past trips and examine what the word 'similar' means mathematically.