Preprocessing Without Data Leakage
Standardizing a feature sounds harmless, but even preprocessing can leak information.
Preprocessing Without Data Leakage explained visually
Standardizing a feature sounds harmless, but even preprocessing can leak information. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Standardizing a feature sounds harmless, but even preprocessing can leak information.
Fit transformations on training data
Standardizing a feature sounds harmless, but even preprocessing can leak information. If the mean and standard deviation are calculated from the entire dataset before splitting, the training workflow has already learned something about validation and test observations. The safer approach fits learned transformations using training data only.
Apply the same rules later
A preprocessing pipeline keeps the transformation and model together. It may fill missing values, encode categories, and scale numerical features. Each fitted step stores the information it learned from training, then applies that same information to later examples. During cross-validation, those steps must be fitted again within each training fold rather than once on the full dataset.
Treat missing values deliberately
Missing-value treatment should match the problem. Filling with a median can be a useful baseline, and a missingness indicator may preserve information about unavailable measurements. It is not a universal repair. Unknown categories at prediction time also need a defined policy. Save the preprocessing configuration with the model, because a correct model receiving differently processed inputs can make incorrect predictions.
A worked example
If the training values are ten, twenty, and thirty, their mean is twenty. Subtract that training mean from a later value of forty to obtain twenty. Do not recalculate the mean by including the new observation; that would change the transformation used by the trained model.
Worked example
If the training values are 10, 20, and 30, their mean is 20. Subtract that training mean from a later value of 40 to obtain 20. Do not recalculate the mean by including the new observation; that would change the transformation used by the trained model.
Quick check: test your understanding
Try it yourself: Where should an imputer be fitted during five-fold cross-validation?
Check the reasoning
Inside each fold, using only that fold's training portion. It then transforms the held-out fold without learning from it.
Is saving model weights enough when training used learned preprocessing?
No. The preprocessing state and feature definitions must be saved too.
How would you solve the practice case?
Inside each fold, using only that fold's training portion. It then transforms the held-out fold without learning from it.
What does the worked example show?
If the training values are 10, 20, and 30, their mean is 20. Subtract that training mean from a later value of 40 to obtain 20. Do not recalculate the mean by including the new observation; that would change the transformation used by the trained model.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 4 transcript210 words · Video transcript · Matches the published narration and captionsOpen transcript
Standardizing a feature sounds harmless, but even preprocessing can leak information. If the mean and standard deviation are calculated from the entire dataset before splitting, the training workflow has already learned something about validation and test observations. The safer approach fits learned transformations using training data only.
A preprocessing pipeline keeps the transformation and model together. It may fill missing values, encode categories, and scale numerical features. Each fitted step stores the information it learned from training, then applies that same information to later examples. During cross-validation, those steps must be fitted again within each training fold rather than once on the full dataset.
Missing-value treatment should match the problem. Filling with a median can be a useful baseline, and a missingness indicator may preserve information about unavailable measurements. It is not a universal repair. Unknown categories at prediction time also need a defined policy. Save the preprocessing configuration with the model, because a correct model receiving differently processed inputs can make incorrect predictions.
If the training values are ten, twenty, and thirty, their mean is twenty. Subtract that training mean from a later value of forty to obtain twenty. Do not recalculate the mean by including the new observation; that would change the transformation used by the trained model.