Training Fit Versus Generalization
A model nearly memorizes every training delivery.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
A model nearly memorizes every training delivery.
A practical question
The model gets almost every training delivery right. Next week, its warnings are much worse. It learned the observed records, but that does not establish how it handles new trips.
The useful idea
Training fit is performance on the examples used to choose model parameters. Generalization concerns new examples from the intended operating setting. A flexible method can fit accidents of one sample, including one-off road events or mistaken labels. The evaluation split determines what future claim we can reasonably make.
Work through the example
Compare errors on training trips with a later held-out week and a new route. Inspect which conditions changed. If performance falls, investigate data coverage, leakage, and model complexity before declaring that more tuning solves it. A tree can memorize the training deliveries and still miss the pattern in a later week. Plot training and held-out error as complexity grows. A large gap suggests the model may be fitting details that do not repeat, but the gap alone does not identify the cause. Inspect labels, sample size, and changes in routes. Choose settings on validation data, then preserve the final test period as a check on the whole development process. Now check the idea on a concrete example: Compare training error with error on a later held-out week. A possible trap is worth naming: A tiny training error does not guarantee useful future decisions. The answer follows from that evidence: Training fit measures performance on seen records; generalization concerns new records from the intended population. Pick a trip the model memorized during training but missed in a later week. Compare its inputs with the nearest training examples and look for a temporary shortcut. The error gap tells you where to investigate, not which exact fix to apply.
Limits and next step
Next, we'll relate the size of our dataset to the flexibility of the rules we ask it to learn.
Worked example
Compare training error with error on a later held-out week.
Quick check: test your understanding
Try it yourself: Consider this case: Compare training error with error on a later held-out week. Explain how you would check the result. In your answer, address this limitation: A tiny training error does not guarantee useful future decisions.
Check the reasoning
Training fit measures performance on seen records; generalization concerns new records from the intended population. A tiny training error does not guarantee useful future decisions.
What is the main idea behind training fit versus generalization?
Training fit measures performance on seen records; generalization concerns new records from the intended population.
How is it used in the example?
Compare training error with error on a later held-out week.
What limitation should be checked?
A tiny training error does not guarantee useful future decisions.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 65 transcript303 words · Narration ready for the upcoming videoOpen transcript
The model gets almost every training delivery right. Next week, its warnings are much worse. It learned the observed records, but that does not establish how it handles new trips.
Training fit is performance on the examples used to choose model parameters. Generalization concerns new examples from the intended operating setting. A flexible method can fit accidents of one sample, including one-off road events or mistaken labels. The evaluation split determines what future claim we can reasonably make.
Compare errors on training trips with a later held-out week and a new route. Inspect which conditions changed. If performance falls, investigate data coverage, leakage, and model complexity before declaring that more tuning solves it. A tree can memorize the training deliveries and still miss the pattern in a later week. Plot training and held-out error as complexity grows. A large gap suggests the model may be fitting details that do not repeat, but the gap alone does not identify the cause. Inspect labels, sample size, and changes in routes. Choose settings on validation data, then preserve the final test period as a check on the whole development process. Now check the idea on a concrete example: Compare training error with error on a later held-out week. A possible trap is worth naming: A tiny training error does not guarantee useful future decisions. The answer follows from that evidence: Training fit measures performance on seen records; generalization concerns new records from the intended population. Pick a trip the model memorized during training but missed in a later week. Compare its inputs with the nearest training examples and look for a temporary shortcut. The error gap tells you where to investigate, not which exact fix to apply.
Next, we'll relate the size of our dataset to the flexibility of the rules we ask it to learn.