Sample Size and Hypothesis Complexity
The team has a small dataset and many flexible candidate rules.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The team has a small dataset and many flexible candidate rules.
A practical question
The dispatcher has only a few hundred labeled deliveries, yet an unconstrained tree can carve them into dozens of tiny leaves. A richer rule set can fit more patterns, including noise. It often needs more representative data for reliable evaluation.
The useful idea
There isn't one universal minimum sample count. What we need depends on the hypothesis family, target error, desired confidence, rarity of severe delays, and assumptions about how examples are sampled. A large dataset made of duplicate trips is not equivalent to diverse routes.
Work through the example
Train a shallow and a deep tree on successively larger valid samples. Compare learning curves on separate future trips. Does the deep tree's validation gap shrink as real coverage grows, or does it remain unstable on a rural subgroup? With a small sample, a highly flexible tree can make a separate rule for nearly every unusual trip. As more independent examples arrive, compare how a shallow and deep tree behave on later deliveries. Do not assume that simply collecting duplicates adds the same evidence as new conditions. Report how many routes and weather types the sample covers. Model capacity and data diversity must be judged together, especially before applying the result to a new depot. Use this case to test your understanding: Compare a shallow tree and an unconstrained tree as sample size grows. Here is the limitation to watch for: There is no universal sample count that makes every model safe. This supports a careful conclusion: More complex hypothesis spaces often need more representative examples to estimate performance reliably. When the dataset grows, count how many genuinely new routes and weather conditions were added. Hundreds of repeated observations from one highway can leave another depot poorly represented. Compare model complexity against the diversity of future cases.
Limits and next step
Next, we'll use a learning-theory thought model to state what a guarantee about future error would actually assume.
Worked example
Compare a shallow tree and an unconstrained tree as sample size grows.
Quick check: test your understanding
Try it yourself: Consider this case: Compare a shallow tree and an unconstrained tree as sample size grows. Explain how you would check the result. In your answer, address this limitation: There is no universal sample count that makes every model safe.
Check the reasoning
More complex hypothesis spaces often need more representative examples to estimate performance reliably. There is no universal sample count that makes every model safe.
What is the main idea behind sample size and hypothesis complexity?
More complex hypothesis spaces often need more representative examples to estimate performance reliably.
How is it used in the example?
Compare a shallow tree and an unconstrained tree as sample size grows.
What limitation should be checked?
There is no universal sample count that makes every model safe.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 66 transcript308 words · Narration ready for the upcoming videoOpen transcript
The dispatcher has only a few hundred labeled deliveries, yet an unconstrained tree can carve them into dozens of tiny leaves. A richer rule set can fit more patterns, including noise. It often needs more representative data for reliable evaluation.
There isn't one universal minimum sample count. What we need depends on the hypothesis family, target error, desired confidence, rarity of severe delays, and assumptions about how examples are sampled. A large dataset made of duplicate trips is not equivalent to diverse routes.
Train a shallow and a deep tree on successively larger valid samples. Compare learning curves on separate future trips. Does the deep tree's validation gap shrink as real coverage grows, or does it remain unstable on a rural subgroup? With a small sample, a highly flexible tree can make a separate rule for nearly every unusual trip. As more independent examples arrive, compare how a shallow and deep tree behave on later deliveries. Do not assume that simply collecting duplicates adds the same evidence as new conditions. Report how many routes and weather types the sample covers. Model capacity and data diversity must be judged together, especially before applying the result to a new depot. Use this case to test your understanding: Compare a shallow tree and an unconstrained tree as sample size grows. Here is the limitation to watch for: There is no universal sample count that makes every model safe. This supports a careful conclusion: More complex hypothesis spaces often need more representative examples to estimate performance reliably. When the dataset grows, count how many genuinely new routes and weather conditions were added. Hundreds of repeated observations from one highway can leave another depot poorly represented. Compare model complexity against the diversity of future cases.
Next, we'll use a learning-theory thought model to state what a guarantee about future error would actually assume.