Monitor Shift and Calibration
The live score distribution shifts after a schedule change.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The live score distribution shifts after a schedule change.
A practical question
After a schedule change, the model starts assigning higher risk scores to many ordinary trips. That change may signal a new input distribution, but it doesn't tell us by itself whether the warnings are wrong.
The useful idea
Track input ranges, missing fields, score distributions, and later true outcomes. Arrival labels come with a delay, so input drift can be visible before we can measure new accuracy or calibration. Compare predicted risk groups with realized delay rates once enough outcomes arrive.
Work through the example
Investigate the schedule change before retraining automatically. It may be a genuine new traffic pattern, a data pipeline error, or a changed definition of lateness. Keep timestamps and examples of the shift. A risk score that once matched delay rates can drift as schedules, routes, or traffic change. Group recent predictions into score ranges and compare them with realized delays after outcomes become known. If there are too few completed trips, wait or state the uncertainty. Also check input ranges and error categories, because a calibration change may be only one symptom. Define who reviews an alert and what evidence is needed before retuning the model. Now check the idea on a concrete example: Compare recent risk bins with realized delay rates when enough outcomes are known. A possible trap is worth naming: A changed input histogram alone does not prove the model has failed. The answer follows from that evidence: Monitor input shift, outcome performance once labels arrive, and probability calibration. Recent risk scores can shift before enough deliveries finish to reveal their true outcomes. Track score distribution now, then compare predicted and realized delay rates when labels arrive. Do not call the model calibrated or broken from unfinished trips.
Limits and next step
Next, we'll decide how to replace a model only when a candidate has enough evidence and a safe rollback path.
Worked example
Compare recent risk bins with realized delay rates when enough outcomes are known.
Quick check: test your understanding
Try it yourself: Consider this case: Compare recent risk bins with realized delay rates when enough outcomes are known. Explain how you would check the result. In your answer, address this limitation: A changed input histogram alone does not prove the model has failed.
Check the reasoning
Monitor input shift, outcome performance once labels arrive, and probability calibration. A changed input histogram alone does not prove the model has failed.
What is the main idea behind monitor shift and calibration?
Monitor input shift, outcome performance once labels arrive, and probability calibration.
How is it used in the example?
Compare recent risk bins with realized delay rates when enough outcomes are known.
What limitation should be checked?
A changed input histogram alone does not prove the model has failed.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 75 transcript299 words · Narration ready for the upcoming videoOpen transcript
After a schedule change, the model starts assigning higher risk scores to many ordinary trips. That change may signal a new input distribution, but it doesn't tell us by itself whether the warnings are wrong.
Track input ranges, missing fields, score distributions, and later true outcomes. Arrival labels come with a delay, so input drift can be visible before we can measure new accuracy or calibration. Compare predicted risk groups with realized delay rates once enough outcomes arrive.
Investigate the schedule change before retraining automatically. It may be a genuine new traffic pattern, a data pipeline error, or a changed definition of lateness. Keep timestamps and examples of the shift. A risk score that once matched delay rates can drift as schedules, routes, or traffic change. Group recent predictions into score ranges and compare them with realized delays after outcomes become known. If there are too few completed trips, wait or state the uncertainty. Also check input ranges and error categories, because a calibration change may be only one symptom. Define who reviews an alert and what evidence is needed before retuning the model. Now check the idea on a concrete example: Compare recent risk bins with realized delay rates when enough outcomes are known. A possible trap is worth naming: A changed input histogram alone does not prove the model has failed. The answer follows from that evidence: Monitor input shift, outcome performance once labels arrive, and probability calibration. Recent risk scores can shift before enough deliveries finish to reveal their true outcomes. Track score distribution now, then compare predicted and realized delay rates when labels arrive. Do not call the model calibrated or broken from unfinished trips.
Next, we'll decide how to replace a model only when a candidate has enough evidence and a safe rollback path.