Split and Train Reproducibly
The team needs another person to repeat its training run.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The team needs another person to repeat its training run.
A practical question
A teammate tries to repeat our validation score and gets a different answer. Which data version, split, or preprocessing setting did we forget to record?
The useful idea
Freeze a time-aware or group-aware split that matches deployment. Fit imputers and scalers on training records alone, then save those fitted objects with the model. Record code version, parameters, label definition, and relevant random seeds. Use the exact same transformation path for validation, test, and future predictions.
Work through the example
Ask the teammate to start from the preserved input data and reproduce a few individual warnings, not just the aggregate score. If the results differ, compare intermediate feature values and timestamps. A clean rebuild should recover the validation result from saved data, split rules, feature code, random settings, and model version. Give the recipe to a second teammate and compare not only the final number but a few individual predictions. If the score changes, inspect data ordering or preprocessing fitted on the wrong period. Reproducibility lets the team distinguish a genuine model improvement from an accidental change in the experiment. Use this case to test your understanding: Rebuild the validation score from a clean environment using the saved recipe. Here is the limitation to watch for: A seed alone cannot reproduce missing data or undocumented features. This supports a careful conclusion: Version the data split, preprocessing, code, parameters, and random seeds. When the second teammate gets a different score, compare saved trip identifiers and preprocessing outputs before tuning anything. The first mismatch in the pipeline is more informative than the difference between two final percentages. Once the inputs and transformations match, compare random settings and library versions. Save the findings so the next training run starts from one agreed recipe.
Limits and next step
Next, we'll compare candidate models under this common protocol and reserve the final test for one honest assessment.
Worked example
Rebuild the validation score from a clean environment using the saved recipe.
Quick check: test your understanding
Try it yourself: Consider this case: Rebuild the validation score from a clean environment using the saved recipe. Explain how you would check the result. In your answer, address this limitation: A seed alone cannot reproduce missing data or undocumented features.
Check the reasoning
Version the data split, preprocessing, code, parameters, and random seeds. A seed alone cannot reproduce missing data or undocumented features.
What is the main idea behind split and train reproducibly?
Version the data split, preprocessing, code, parameters, and random seeds.
How is it used in the example?
Rebuild the validation score from a clean environment using the saved recipe.
What limitation should be checked?
A seed alone cannot reproduce missing data or undocumented features.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 78 transcript300 words · Narration ready for the upcoming videoOpen transcript
A teammate tries to repeat our validation score and gets a different answer. Which data version, split, or preprocessing setting did we forget to record?
Freeze a time-aware or group-aware split that matches deployment. Fit imputers and scalers on training records alone, then save those fitted objects with the model. Record code version, parameters, label definition, and relevant random seeds. Use the exact same transformation path for validation, test, and future predictions.
Ask the teammate to start from the preserved input data and reproduce a few individual warnings, not just the aggregate score. If the results differ, compare intermediate feature values and timestamps. A clean rebuild should recover the validation result from saved data, split rules, feature code, random settings, and model version. Give the recipe to a second teammate and compare not only the final number but a few individual predictions. If the score changes, inspect data ordering or preprocessing fitted on the wrong period. Reproducibility lets the team distinguish a genuine model improvement from an accidental change in the experiment. Use this case to test your understanding: Rebuild the validation score from a clean environment using the saved recipe. Here is the limitation to watch for: A seed alone cannot reproduce missing data or undocumented features. This supports a careful conclusion: Version the data split, preprocessing, code, parameters, and random seeds. When the second teammate gets a different score, compare saved trip identifiers and preprocessing outputs before tuning anything. The first mismatch in the pipeline is more informative than the difference between two final percentages. Once the inputs and transformations match, compare random settings and library versions. Save the findings so the next training run starts from one agreed recipe.
Next, we'll compare candidate models under this common protocol and reserve the final test for one honest assessment.