Regularization and Cross-Validation
A flexible model can explain the training data in many ways, including ways that follow noise.
Regularization and Cross-Validation explained visually
A flexible model can explain the training data in many ways, including ways that follow noise. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A flexible model can explain the training data in many ways, including ways that follow noise.
Control flexibility
A flexible model can explain the training data in many ways, including ways that follow noise. Regularization constrains that flexibility. It may penalize large coefficients, restrict tree complexity, or otherwise favor simpler solutions. The goal is better performance on new data, not merely a smaller training error.
Repeat a fair comparison
Ridge regression penalizes squared coefficient sizes, while lasso uses an absolute-value penalty that can set some coefficients to zero. Feature scaling matters when penalties compare coefficient magnitudes across differently measured inputs. The penalty strength is a model-selection choice. Too little may leave overfitting; too much may remove useful signal.
Keep every fitted step inside the fold
Cross-validation compares choices across several training-and-validation partitions. Use grouped or time-aware folds when the observations require them. Fit preprocessing separately inside each training fold. After selecting settings, refit the chosen pipeline on the appropriate development data, then evaluate on the untouched test set. Report variation across folds as context, and remember that cross-validation cannot compensate for an evaluation design that does not resemble deployment.
A worked example
In five-fold cross-validation, each example serves in a held-out fold once while the model is fitted on the other four folds. If a feature scaler is fitted on all five folds before this process, information from the held-out fold leaks into every evaluation.
Worked example
In five-fold cross-validation, each example serves in a held-out fold once while the model is fitted on the other four folds. If a feature scaler is fitted on all five folds before this process, information from the held-out fold leaks into every evaluation.
Quick check: test your understanding
Try it yourself: You compare two penalty strengths with nearly identical average validation errors. What else might guide the choice?
Check the reasoning
Consider variation across folds, simplicity, stability, computational cost, and the application's error requirements rather than assuming a tiny average difference is decisive.
Does stronger regularization always improve a model?
No. Excessive regularization can underfit useful relationships.
How would you solve the practice case?
Consider variation across folds, simplicity, stability, computational cost, and the application's error requirements rather than assuming a tiny average difference is decisive.
What does the worked example show?
In five-fold cross-validation, each example serves in a held-out fold once while the model is fitted on the other four folds. If a feature scaler is fitted on all five folds before this process, information from the held-out fold leaks into every evaluation.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 11 transcript203 words · Video transcript · Matches the published narration and captionsOpen transcript
A flexible model can explain the training data in many ways, including ways that follow noise. Regularization constrains that flexibility. It may penalize large coefficients, restrict tree complexity, or otherwise favor simpler solutions. The goal is better performance on new data, not merely a smaller training error.
Ridge regression penalizes squared coefficient sizes, while lasso uses an absolute-value penalty that can set some coefficients to zero. Feature scaling matters when penalties compare coefficient magnitudes across differently measured inputs. The penalty strength is a model-selection choice. Too little may leave overfitting; too much may remove useful signal.
Cross-validation compares choices across several training-and-validation partitions. Use grouped or time-aware folds when the observations require them. Fit preprocessing separately inside each training fold. After selecting settings, refit the chosen pipeline on the appropriate development data, then evaluate on the untouched test set. Report variation across folds as context, and remember that cross-validation cannot compensate for an evaluation design that does not resemble deployment.
In five-fold cross-validation, each example serves in a held-out fold once while the model is fitted on the other four folds. If a feature scaler is fitted on all five folds before this process, information from the held-out fold leaks into every evaluation.