Evaluate by Conditions and Subgroups
The system works by day but misses spaces at night.
Read the full lesson and open its transcript below. A narrated video will be added after production.
What you will learn
The system works by day but misses spaces at night.
A practical question
The parking model scores well overall, but it misses spaces under a particular tree every afternoon. An aggregate number can bury that pattern.
The useful idea
Group predictions by light, weather, camera, distance, and any other condition relevant to the workflow. Compare missed occupied spaces and false occupied calls within each group. Smaller groups have noisier estimates, so show counts alongside rates.
Work through the example
Open representative mistakes and check whether the ground truth itself is uncertain. If the tree shadow causes a repeated failure, collect honest examples and evaluate fixes on separate data. Don't repeatedly tune on the final held-out set. A single accuracy figure can hide poor results in rain or at night. Break the held-out set into conditions that the garage will actually face and report both errors and sample counts. If a group contains only a few examples, say that the estimate is uncertain. Inspect why the errors differ before prescribing a fix; changing the threshold might help one condition and harm another. Keep the final evaluation set separate from repeated tuning. Use this case to test your understanding: Report occupancy accuracy separately for daylight, rain, and low light. Here is the limitation to watch for: A small subgroup sample can make an apparent gap unstable. The answer follows from that evidence: Evaluate errors by relevant conditions and groups, along with counts and uncertainty. A model can look strong overall because most frames were filmed in easy daylight. Show how many rainy and night scenes were tested and which spaces were missed. A small subgroup needs more evidence before a confident claim. Compare mistakes in each condition at the actual stalls, since one missed occupied space can affect a visitor despite a strong overall average.
Limits and next step
The next challenge is a camera we haven't used before. We'll test whether the system's learned cues transfer to that new view.
Worked example
Report occupancy accuracy separately for daylight, rain, and low light.
Quick check: test your understanding
Try it yourself: Consider this case: Report occupancy accuracy separately for daylight, rain, and low light. Explain how you would check the result. In your answer, address this limitation: A small subgroup sample can make an apparent gap unstable.
Check the reasoning
Evaluate errors by relevant conditions and groups, along with counts and uncertainty. A small subgroup sample can make an apparent gap unstable.
What is the main idea behind evaluate by conditions and subgroups?
Evaluate errors by relevant conditions and groups, along with counts and uncertainty.
How is it used in the example?
Report occupancy accuracy separately for daylight, rain, and low light.
What limitation should be checked?
A small subgroup sample can make an apparent gap unstable.
Further reading
Programming Computer Vision with Python — Jan Erik Solem.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 71 transcript305 words · Narration ready for the upcoming videoOpen transcript
The parking model scores well overall, but it misses spaces under a particular tree every afternoon. An aggregate number can bury that pattern.
Group predictions by light, weather, camera, distance, and any other condition relevant to the workflow. Compare missed occupied spaces and false occupied calls within each group. Smaller groups have noisier estimates, so show counts alongside rates.
Open representative mistakes and check whether the ground truth itself is uncertain. If the tree shadow causes a repeated failure, collect honest examples and evaluate fixes on separate data. Don't repeatedly tune on the final held-out set. A single accuracy figure can hide poor results in rain or at night. Break the held-out set into conditions that the garage will actually face and report both errors and sample counts. If a group contains only a few examples, say that the estimate is uncertain. Inspect why the errors differ before prescribing a fix; changing the threshold might help one condition and harm another. Keep the final evaluation set separate from repeated tuning. Use this case to test your understanding: Report occupancy accuracy separately for daylight, rain, and low light. Here is the limitation to watch for: A small subgroup sample can make an apparent gap unstable. The answer follows from that evidence: Evaluate errors by relevant conditions and groups, along with counts and uncertainty. A model can look strong overall because most frames were filmed in easy daylight. Show how many rainy and night scenes were tested and which spaces were missed. A small subgroup needs more evidence before a confident claim. Compare mistakes in each condition at the actual stalls, since one missed occupied space can affect a visitor despite a strong overall average.
The next challenge is a camera we haven't used before. We'll test whether the system's learned cues transfer to that new view.