Compare Candidates on Honest Data
Several candidate models look close on validation data.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
Several candidate models look close on validation data.
A practical question
Several delivery models perform similarly on validation weeks. Choose among them using the same split, target, preprocessing rules, and operating metrics. Tune thresholds there too, then freeze the selection.
The useful idea
Only after those choices are fixed should we run the selected system on an untouched later test period. Report missed severe delays, false warning workload, subgroup results, and uncertainty around estimates. Open the cases where competing models disagree so a small average gain doesn't hide a costly failure.
Work through the example
If you return to tune after seeing the final test, you will need new untouched evidence for another final claim. The test is a measurement of a frozen process, not an endless scoreboard. Put candidate methods on the same held-out comparison with the same target and available inputs. Report uncertainty in the score and show meaningful mistakes, such as missed severe delays and excess warnings for one depot. If the apparent winner depends on a tiny set of examples, gather more evidence before changing operations. Keep the final test period separate from the choices made during development. A leaderboard alone is not a deployment decision. Now check the idea on a concrete example: Report both score uncertainty and important error categories. A possible trap is worth naming: Repeated test-set tuning turns the final check into more validation. The answer follows from that evidence: Select on the same validation protocol, then assess the chosen candidate on an untouched test period. Two candidates can have nearly equal average scores but different severe misses. Put those mistakes and their sample counts beside the uncertainty interval. If the evidence is too close to call, keep the simpler baseline while gathering a more informative future period.
Limits and next step
Next, we'll turn those results into a deployment plan that states both benefits and limits.
Worked example
Report both score uncertainty and important error categories.
Quick check: test your understanding
Try it yourself: Consider this case: Report both score uncertainty and important error categories. Explain how you would check the result. In your answer, address this limitation: Repeated test-set tuning turns the final check into more validation.
Check the reasoning
Select on the same validation protocol, then assess the chosen candidate on an untouched test period. Repeated test-set tuning turns the final check into more validation.
What is the main idea behind compare candidates on honest data?
Select on the same validation protocol, then assess the chosen candidate on an untouched test period.
How is it used in the example?
Report both score uncertainty and important error categories.
What limitation should be checked?
Repeated test-set tuning turns the final check into more validation.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 79 transcript293 words · Narration ready for the upcoming videoOpen transcript
Several delivery models perform similarly on validation weeks. Choose among them using the same split, target, preprocessing rules, and operating metrics. Tune thresholds there too, then freeze the selection.
Only after those choices are fixed should we run the selected system on an untouched later test period. Report missed severe delays, false warning workload, subgroup results, and uncertainty around estimates. Open the cases where competing models disagree so a small average gain doesn't hide a costly failure.
If you return to tune after seeing the final test, you will need new untouched evidence for another final claim. The test is a measurement of a frozen process, not an endless scoreboard. Put candidate methods on the same held-out comparison with the same target and available inputs. Report uncertainty in the score and show meaningful mistakes, such as missed severe delays and excess warnings for one depot. If the apparent winner depends on a tiny set of examples, gather more evidence before changing operations. Keep the final test period separate from the choices made during development. A leaderboard alone is not a deployment decision. Now check the idea on a concrete example: Report both score uncertainty and important error categories. A possible trap is worth naming: Repeated test-set tuning turns the final check into more validation. The answer follows from that evidence: Select on the same validation protocol, then assess the chosen candidate on an untouched test period. Two candidates can have nearly equal average scores but different severe misses. Put those mistakes and their sample counts beside the uncertainty interval. If the evidence is too close to call, keep the simpler baseline while gathering a more informative future period.
Next, we'll turn those results into a deployment plan that states both benefits and limits.