Retrain and Roll Back with Evidence
A replacement model beats the old one in one week.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
A replacement model beats the old one in one week.
A practical question
A new delay model wins on one recent week. Should it replace the old model for every depot? First define what success means over several conditions and how we would reverse the change if errors rise.
The useful idea
Version the training data, feature definitions, thresholds, code, and fitted model. Run the candidate in shadow mode or on a limited share of decisions while comparing both versions on the same future trips. Check rural routes, severe delays, review load, and calibration.
Work through the example
Set rollback criteria before rollout and keep the old pipeline ready. A model file alone isn't enough if its input transformations have changed. Save evidence about why the replacement was chosen. Keep the current model running while a proposed replacement is evaluated on the same future period. Compare performance by route condition and review workload, not only one average metric. Decide in advance what improvement justifies switching and what failure calls for rollback. Save the old model, preprocessing code, and threshold so reverting is practical. A deployment record should show which version produced each warning and why the team changed it. Use this case to test your understanding: Run both versions on the same future period before switching all traffic. Here is the limitation to watch for: A single favorable week may be noise or a temporary condition. This supports a careful conclusion: Use staged evaluation, versioned features, a rollback plan, and evidence across conditions. A new version may win during one quiet week and lose under heavy rain. Evaluate both versions on shared future conditions, document the change rule, and keep the old package ready. That makes a rollback possible without guessing which code produced an earlier alert. Record the switching date as well.
Limits and next step
Now we'll assemble a final delivery-delay project from the decision question onward.
Worked example
Run both versions on the same future period before switching all traffic.
Quick check: test your understanding
Try it yourself: Consider this case: Run both versions on the same future period before switching all traffic. Explain how you would check the result. In your answer, address this limitation: A single favorable week may be noise or a temporary condition.
Check the reasoning
Use staged evaluation, versioned features, a rollback plan, and evidence across conditions. A single favorable week may be noise or a temporary condition.
What is the main idea behind retrain and roll back with evidence?
Use staged evaluation, versioned features, a rollback plan, and evidence across conditions.
How is it used in the example?
Run both versions on the same future period before switching all traffic.
What limitation should be checked?
A single favorable week may be noise or a temporary condition.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 76 transcript295 words · Narration ready for the upcoming videoOpen transcript
A new delay model wins on one recent week. Should it replace the old model for every depot? First define what success means over several conditions and how we would reverse the change if errors rise.
Version the training data, feature definitions, thresholds, code, and fitted model. Run the candidate in shadow mode or on a limited share of decisions while comparing both versions on the same future trips. Check rural routes, severe delays, review load, and calibration.
Set rollback criteria before rollout and keep the old pipeline ready. A model file alone isn't enough if its input transformations have changed. Save evidence about why the replacement was chosen. Keep the current model running while a proposed replacement is evaluated on the same future period. Compare performance by route condition and review workload, not only one average metric. Decide in advance what improvement justifies switching and what failure calls for rollback. Save the old model, preprocessing code, and threshold so reverting is practical. A deployment record should show which version produced each warning and why the team changed it. Use this case to test your understanding: Run both versions on the same future period before switching all traffic. Here is the limitation to watch for: A single favorable week may be noise or a temporary condition. This supports a careful conclusion: Use staged evaluation, versioned features, a rollback plan, and evidence across conditions. A new version may win during one quiet week and lose under heavy rain. Evaluate both versions on shared future conditions, document the change rule, and keep the old package ready. That makes a rollback possible without guessing which code produced an earlier alert. Record the switching date as well.
Now we'll assemble a final delivery-delay project from the decision question onward.