Recognize When the World Changes
A highway closes and travel patterns change suddenly.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
A highway closes and travel patterns change suddenly.
A practical question
A highway closes for repairs. Suddenly the journey patterns that trained our delay model no longer describe many planned trips. This is distribution shift: future examples differ from those used to estimate performance.
The useful idea
Monitor input values such as routes, distances, and closure flags. As actual arrival labels become available, compare error and calibration before and after the change. A shift in features is an early clue, not proof that the model is already making more mistakes.
Work through the example
If the closure schedule is known before departure, add it to the operational information or use a temporary reviewed rule. Evaluate any retrained model on post-change trips held aside for checking. Keep the earlier version and its context. A road closure changes which routes are used and how long they take. Compare feature ranges and realized delay rates before and after it began. If the old model's errors rise, identify whether the input population changed, the relationship to the label changed, or both. Keep a timestamped baseline and a review trigger. A changing chart calls for investigation; blindly retraining on a few recent trips can make the system less stable. Use this case to test your understanding: Compare delay rates and feature ranges before and after the closure. Here is the limitation to watch for: An old score cannot certify a model under a changed world. This supports a careful conclusion: Distribution shift means future examples can differ from the data on which performance was estimated. Mark the week the closure began and compare both input ranges and errors on either side. If the labels themselves were recorded differently, retraining alone will not repair the measurement. Keep a human review route while the change is understood.
Limits and next step
Next, we'll move from predicting a delay to choosing an action, where the decision itself can affect what happens later.
Worked example
Compare delay rates and feature ranges before and after the closure.
Quick check: test your understanding
Try it yourself: Consider this case: Compare delay rates and feature ranges before and after the closure. Explain how you would check the result. In your answer, address this limitation: An old score cannot certify a model under a changed world.
Check the reasoning
Distribution shift means future examples can differ from the data on which performance was estimated. An old score cannot certify a model under a changed world.
What is the main idea behind recognize when the world changes?
Distribution shift means future examples can differ from the data on which performance was estimated.
How is it used in the example?
Compare delay rates and feature ranges before and after the closure.
What limitation should be checked?
An old score cannot certify a model under a changed world.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 68 transcript302 words · Narration ready for the upcoming videoOpen transcript
A highway closes for repairs. Suddenly the journey patterns that trained our delay model no longer describe many planned trips. This is distribution shift: future examples differ from those used to estimate performance.
Monitor input values such as routes, distances, and closure flags. As actual arrival labels become available, compare error and calibration before and after the change. A shift in features is an early clue, not proof that the model is already making more mistakes.
If the closure schedule is known before departure, add it to the operational information or use a temporary reviewed rule. Evaluate any retrained model on post-change trips held aside for checking. Keep the earlier version and its context. A road closure changes which routes are used and how long they take. Compare feature ranges and realized delay rates before and after it began. If the old model's errors rise, identify whether the input population changed, the relationship to the label changed, or both. Keep a timestamped baseline and a review trigger. A changing chart calls for investigation; blindly retraining on a few recent trips can make the system less stable. Use this case to test your understanding: Compare delay rates and feature ranges before and after the closure. Here is the limitation to watch for: An old score cannot certify a model under a changed world. This supports a careful conclusion: Distribution shift means future examples can differ from the data on which performance was estimated. Mark the week the closure began and compare both input ranges and errors on either side. If the labels themselves were recorded differently, retraining alone will not repair the measurement. Keep a human review route while the change is understood.
Next, we'll move from predicting a delay to choosing an action, where the decision itself can affect what happens later.