Compare Two Models Without Test-Set Tuning
Two models differ by one percentage point on the held-out period.
Compare Two Models Without Test-Set Tuning explained visually
Two models differ by one percentage point on the held-out period. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Two models differ by one percentage point on the held-out period.
A practical question
Model A catches one percent more delayed trips than Model B in a held-out week. Is that enough to replace the current system? First ask how many actual trips differ and what those differences cost.
The useful idea
Choose model settings using validation data. Then evaluate both candidates on the same final test deliveries so each faces identical cases. Examine where their warnings disagree, and report uncertainty around the difference. One point may be smaller than the variation across weeks.
Work through the example
Decide the operational improvement that matters before peeking at the final scores. Repeatedly selecting a new model based on that same test period turns it into another tuning set.
Limits and next step
Next, we'll change the target from a late-or-not warning to a numeric estimate of minutes late.
Worked example
Report the difference alongside its uncertainty and the cases where predictions disagree.
Quick check: test your understanding
Try it yourself: In the situation described above, explain the method and one reason its output might be misleading.
Check the reasoning
Choose candidates with validation data, then compare final outcomes on the same untouched test set. Repeatedly adjusting after seeing test results makes the comparison optimistic.
What is the main idea behind compare two models without test-set tuning?
Choose candidates with validation data, then compare final outcomes on the same untouched test set.
How is it used in the example?
Report the difference alongside its uncertainty and the cases where predictions disagree.
What limitation should be checked?
Repeatedly adjusting after seeing test results makes the comparison optimistic.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 36 transcript122 words · Video transcript · Matches the published narration and captionsOpen transcript
Model A catches one percent more delayed trips than Model B in a held-out week. Is that enough to replace the current system? First ask how many actual trips differ and what those differences cost.
Choose model settings using validation data. Then evaluate both candidates on the same final test deliveries so each faces identical cases. Examine where their warnings disagree, and report uncertainty around the difference. One point may be smaller than the variation across weeks.
Decide the operational improvement that matters before peeking at the final scores. Repeatedly selecting a new model based on that same test period turns it into another tuning set.
Next, we'll change the target from a late-or-not warning to a numeric estimate of minutes late.