Training, Validation, and Test Data
Our dispatcher has a thousand past deliveries and a clear task: flag a likely delay before departure.
Training, Validation, and Test Data explained visually
Our dispatcher has a thousand past deliveries and a clear task: flag a likely delay before departure. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Our dispatcher has a thousand past deliveries and a clear task: flag a likely delay before departure.
Past and future deliveries
Our dispatcher has one thousand past deliveries. We want to learn from them and then judge whether a delay warning will work on future trips. For that, we need three separate roles for the data: training, validation, and testing.
Training, validation, and test
Training records help a model learn its parameters. Validation records help us choose settings and compare the model with the simple route-based baseline. Test records are saved for a final assessment after those choices. If we repeatedly change the model because of its test score, the test period stops being an independent check.
Split in time
Because the real system will face later deliveries, we can split the records in time order. Use the earliest seven hundred for training, the next one hundred and fifty for validation, and the final one hundred and fifty for testing. The counts add to one thousand. That arithmetic is easy; the important part is keeping every record from one delivery in one period and using only inputs known before its departure.
Check a difficult week
Suppose unusual weather appears in the final period and the score drops. That is useful evidence about a new condition, not a reason to hide the week by mixing it into training. Record the split, compare candidates on the same validation period, and inspect the final errors. The rule is simple: learn from the past, choose with validation, and check the chosen system once on untouched test data. Next, we'll explore the training records themselves.
Worked example
Suppose 1,000 independent examples are split into 700 training, 150 validation, and 150 test examples. You fit on the 700, choose between candidate settings with the 150 validation examples, and evaluate the chosen system once on the remaining 150. Related observations require a suitable grouped or temporal split instead.
Quick check: test your understanding
Try it yourself: How would you split a dataset to predict next month's demand from earlier months?
Check the reasoning
Use a chronological design so training precedes validation and testing. Avoid giving the model measurements from the future of a prediction point.
Which set fits the model parameters?
Training data.
Which set selects settings before final assessment?
Validation data.
With 700 train and 150 validation out of 1,000, how many test records remain?
150 records, held aside for the final check.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 2 transcript238 words · Video transcript · Matches the published narration and captionsOpen transcript
Our dispatcher has one thousand past deliveries. We want to learn from them and then judge whether a delay warning will work on future trips. For that, we need three separate roles for the data: training, validation, and testing.
Training records help a model learn its parameters. Validation records help us choose settings and compare the model with the simple route-based baseline. Test records are saved for a final assessment after those choices. If we repeatedly change the model because of its test score, the test period stops being an independent check.
Because the real system will face later deliveries, we can split the records in time order. Use the earliest seven hundred for training, the next one hundred and fifty for validation, and the final one hundred and fifty for testing. The counts add to one thousand. That arithmetic is easy; the important part is keeping every record from one delivery in one period and using only inputs known before its departure.
Suppose unusual weather appears in the final period and the score drops. That is useful evidence about a new condition, not a reason to hide the week by mixing it into training. Record the split, compare candidates on the same validation period, and inspect the final errors. The rule is simple: learn from the past, choose with validation, and check the chosen system once on untouched test data. Next, we'll explore the training records themselves.