Tune a Review Threshold
A doubtful space prediction could trigger an unnecessary warning.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
A doubtful space prediction could trigger an unnecessary warning.
A practical question
The garage has one operator available for uncertain cases. If we send every moderately uncertain frame to review, the queue grows faster than anyone can clear it. If we review almost nothing, consequential mistakes go unchecked.
The useful idea
On held-out data, compare error rates and review volume at several thresholds. Decide separately what to do with likely empty, likely occupied, and uncertain states. A confidence value should be checked for calibration under each important camera condition.
Work through the example
Estimate how many cases per hour the operator would see and open examples at the boundary. When the camera moves or lighting changes, that workload may change too. Keep a safe fallback when no reviewer is available. A lower threshold can send more ambiguous spaces to review, but the team has a limited capacity. Compare error rates and the number of daily reviews for several settings on validation scenes. Include a difficult condition such as glare rather than choosing on easy daylight alone. Set the rule before looking at the final test. If review demand exceeds staffing, the answer may require a better sensor or workflow, not a magically perfect threshold. Now check the idea on a concrete example: Compare error rates and review volume at several thresholds on validation data. A possible trap is worth naming: Confidence scores need checking; a numerical score is not automatically calibrated. This supports a careful conclusion: A review threshold trades automated coverage against mistakes requiring human correction. Put review volume beside errors for each candidate threshold. If the best-looking setting sends more spaces than the operator can inspect, revise the workflow or gather better evidence. A threshold cannot create human capacity. Check the queue on busy days before deciding that the new policy is feasible.
Limits and next step
After deployment, thresholds may stop behaving as expected. Next, we'll watch for drift in the camera and its predictions.
Worked example
Compare error rates and review volume at several thresholds on validation data.
Quick check: test your understanding
Try it yourself: Consider this case: Compare error rates and review volume at several thresholds on validation data. Explain how you would check the result. In your answer, address this limitation: Confidence scores need checking; a numerical score is not automatically calibrated.
Check the reasoning
A review threshold trades automated coverage against mistakes requiring human correction. Confidence scores need checking; a numerical score is not automatically calibrated.
What is the main idea behind tune a review threshold?
A review threshold trades automated coverage against mistakes requiring human correction.
How is it used in the example?
Compare error rates and review volume at several thresholds on validation data.
What limitation should be checked?
Confidence scores need checking; a numerical score is not automatically calibrated.
Further reading
Programming Computer Vision with Python — Jan Erik Solem.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 74 transcript305 words · Narration ready for the upcoming videoOpen transcript
The garage has one operator available for uncertain cases. If we send every moderately uncertain frame to review, the queue grows faster than anyone can clear it. If we review almost nothing, consequential mistakes go unchecked.
On held-out data, compare error rates and review volume at several thresholds. Decide separately what to do with likely empty, likely occupied, and uncertain states. A confidence value should be checked for calibration under each important camera condition.
Estimate how many cases per hour the operator would see and open examples at the boundary. When the camera moves or lighting changes, that workload may change too. Keep a safe fallback when no reviewer is available. A lower threshold can send more ambiguous spaces to review, but the team has a limited capacity. Compare error rates and the number of daily reviews for several settings on validation scenes. Include a difficult condition such as glare rather than choosing on easy daylight alone. Set the rule before looking at the final test. If review demand exceeds staffing, the answer may require a better sensor or workflow, not a magically perfect threshold. Now check the idea on a concrete example: Compare error rates and review volume at several thresholds on validation data. A possible trap is worth naming: Confidence scores need checking; a numerical score is not automatically calibrated. This supports a careful conclusion: A review threshold trades automated coverage against mistakes requiring human correction. Put review volume beside errors for each candidate threshold. If the best-looking setting sends more spaces than the operator can inspect, revise the workflow or gather better evidence. A threshold cannot create human capacity. Check the queue on busy days before deciding that the new policy is feasible.
After deployment, thresholds may stop behaving as expected. Next, we'll watch for drift in the camera and its predictions.