Check Whether Probabilities Are Calibrated
Among deliveries given a 70% late risk, far fewer are actually late.
Check Whether Probabilities Are Calibrated explained visually
Among deliveries given a 70% late risk, far fewer are actually late. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Among deliveries given a 70% late risk, far fewer are actually late.
A practical question
The model gives a set of deliveries a seventy-percent late risk. In the held-out data, only about forty percent of those trips are late. The model may rank difficult trips ahead of easy ones, yet it is overconfident about their probabilities.
The useful idea
Calibration compares predicted risk with observed frequency across groups of predictions. Gather enough cases in several score ranges and report their counts. Small groups can fluctuate widely, so don't claim precise calibration from a handful of deliveries.
Work through the example
Plot or tabulate the predicted and observed rates. If the mismatch matters to decisions, fit a calibration step using separate validation data and test it on untouched data. A calibrated risk can still be wrong for one trip; it describes frequencies across comparable cases.
Limits and next step
Next, we'll connect those probabilities to practical mistakes using a confusion matrix.
Worked example
Group validation predictions near 0.7 and compare with their actual late fraction.
Quick check: test your understanding
Try it yourself: In the situation described above, explain the method and one reason its output might be misleading.
Check the reasoning
Calibration asks whether predicted frequencies match observed frequencies on held-out data. Good ranking and good calibration are different qualities.
What is the main idea behind check whether probabilities are calibrated?
Calibration asks whether predicted frequencies match observed frequencies on held-out data.
How is it used in the example?
Group validation predictions near 0.7 and compare with their actual late fraction.
What limitation should be checked?
Good ranking and good calibration are different qualities.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 32 transcript134 words · Video transcript · Matches the published narration and captionsOpen transcript
The model gives a set of deliveries a seventy-percent late risk. In the held-out data, only about forty percent of those trips are late. The model may rank difficult trips ahead of easy ones, yet it is overconfident about their probabilities.
Calibration compares predicted risk with observed frequency across groups of predictions. Gather enough cases in several score ranges and report their counts. Small groups can fluctuate widely, so don't claim precise calibration from a handful of deliveries.
Plot or tabulate the predicted and observed rates. If the mismatch matters to decisions, fit a calibration step using separate validation data and test it on untouched data. A calibrated risk can still be wrong for one trip; it describes frequencies across comparable cases.
Next, we'll connect those probabilities to practical mistakes using a confusion matrix.