Regularize a Flexible Linear Model
Many related route indicators make coefficients unstable.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
Many related route indicators make coefficients unstable.
A practical question
Our delivery model includes distance plus many route indicators that tell similar stories. Coefficients swing when a few training trips change. Regularization adds a penalty during fitting that discourages overly large or complex weights.
The useful idea
An L two penalty generally shrinks weights smoothly. An L one penalty can set some weights to zero under suitable conditions. Neither choice knows which feature is causal. Scale and encoding matter, because a penalty on coefficients interacts with their units.
Work through the example
Try several penalty strengths on validation data and compare held-out prediction errors, not just the neatness of the coefficients. A very strong penalty can erase useful signal; a very weak one may leave the instability untouched. Many route flags describe similar conditions, so a model can assign unstable weights to them while still predicting acceptably. Try a weak and strong penalty on the same training period, then compare later trips. A smoother coefficient table is not the goal; reliable future estimates are. Standardize the numeric inputs using training data before comparing penalty strengths, because units can change how heavily each weight is penalized. Save the scaler and penalty with the model. If every setting misses bad-weather trips, a different feature or data source may matter more. Now check the idea on a concrete example: Tune penalty strength on validation data and compare prediction errors. A possible trap is worth naming: Too much penalty can suppress useful signal. The answer follows from that evidence: Regularization adds a penalty that discourages large or complex weights during fitting. When two route flags carry nearly the same information, the penalty may move weight between them. Compare predictions as well as weights after refitting on a different training sample. The model is useful if the warnings remain dependable, even when individual coefficients shift.
Limits and next step
Record the scaler and penalty with the model. Next, we'll return to a classification score and choose the point where it triggers an actual warning.
Worked example
Tune penalty strength on validation data and compare prediction errors.
Quick check: test your understanding
Try it yourself: Consider this case: Tune penalty strength on validation data and compare prediction errors. Explain how you would check the result. In your answer, address this limitation: Too much penalty can suppress useful signal.
Check the reasoning
Regularization adds a penalty that discourages large or complex weights during fitting. Too much penalty can suppress useful signal.
What is the main idea behind regularize a flexible linear model?
Regularization adds a penalty that discourages large or complex weights during fitting.
How is it used in the example?
Tune penalty strength on validation data and compare prediction errors.
What limitation should be checked?
Too much penalty can suppress useful signal.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 39 transcript318 words · Narration ready for the upcoming videoOpen transcript
Our delivery model includes distance plus many route indicators that tell similar stories. Coefficients swing when a few training trips change. Regularization adds a penalty during fitting that discourages overly large or complex weights.
An L two penalty generally shrinks weights smoothly. An L one penalty can set some weights to zero under suitable conditions. Neither choice knows which feature is causal. Scale and encoding matter, because a penalty on coefficients interacts with their units.
Try several penalty strengths on validation data and compare held-out prediction errors, not just the neatness of the coefficients. A very strong penalty can erase useful signal; a very weak one may leave the instability untouched. Many route flags describe similar conditions, so a model can assign unstable weights to them while still predicting acceptably. Try a weak and strong penalty on the same training period, then compare later trips. A smoother coefficient table is not the goal; reliable future estimates are. Standardize the numeric inputs using training data before comparing penalty strengths, because units can change how heavily each weight is penalized. Save the scaler and penalty with the model. If every setting misses bad-weather trips, a different feature or data source may matter more. Now check the idea on a concrete example: Tune penalty strength on validation data and compare prediction errors. A possible trap is worth naming: Too much penalty can suppress useful signal. The answer follows from that evidence: Regularization adds a penalty that discourages large or complex weights during fitting. When two route flags carry nearly the same information, the penalty may move weight between them. Compare predictions as well as weights after refitting on a different training sample. The model is useful if the warnings remain dependable, even when individual coefficients shift.
Record the scaler and penalty with the model. Next, we'll return to a classification score and choose the point where it triggers an actual warning.