Build a Dataset That Tests Generalization
A model performs brilliantly on held-out video frames, then fails on a new camera.
Build a Dataset That Tests Generalization explained visually
A model performs brilliantly on held-out video frames, then fails on a new camera. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A model performs brilliantly on held-out video frames, then fails on a new camera.
Similarity can inflate results
A model performs brilliantly on held-out video frames, then fails on a new camera. One possible reason is data leakage. If nearly identical neighboring frames appear in both training and testing, the test does not represent a truly independent situation. The files are different, but much of the visual information is shared.
Split by the source of dependence
Choose the split unit according to the intended deployment. It may be a whole video, camera, location, recording day, or individual subject. Keep related samples together when their similarity could leak information. Then use validation data for model selection and reserve the test set for the final assessment. Repeatedly tuning against the test set turns it into another development set.
Review labels and coverage
Coverage matters as much as count. Include the lighting, object sizes, viewpoints, and difficult conditions that the application is expected to handle. Review labels and use duplicate checks as supporting tools, not as proof of independence. Two images from the same scene can be highly related without being exact duplicates. Document the split rule and any missing conditions so that performance claims match the evidence.
A worked example
A one-minute clip produces one,eight hundred frames at thirty frames per second. Randomly assigning frames to train and test can place near-identical moments in both groups. Splitting by complete recording sessions gives a more demanding test of performance on a new session.
Worked example
A one-minute clip produces 1,800 frames at 30 frames per second. Randomly assigning frames to train and test can place near-identical moments in both groups. Splitting by complete recording sessions gives a more demanding test of performance on a new session.
Quick check: test your understanding
Try it yourself: Design a split for a system intended to work on a camera location it has never seen.
Check the reasoning
Reserve one or more complete camera locations for testing, and use separate training and validation locations or groups for development when the available data permits.
Does an exact-duplicate report detect every form of data leakage?
No. Related scenes, subjects, and recording conditions can leak information without identical files.
How would you solve the practice case?
Reserve one or more complete camera locations for testing, and use separate training and validation locations or groups for development when the available data permits.
What does the worked example show?
A one-minute clip produces 1,800 frames at 30 frames per second. Randomly assigning frames to train and test can place near-identical moments in both groups. Splitting by complete recording sessions gives a more demanding test of performance on a new session.
Further reading
Programming Computer Vision with Python — Jan Erik Solem.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 12 transcript219 words · Video transcript · Matches the published narration and captionsOpen transcript
A model performs brilliantly on held-out video frames, then fails on a new camera. One possible reason is data leakage. If nearly identical neighboring frames appear in both training and testing, the test does not represent a truly independent situation. The files are different, but much of the visual information is shared.
Choose the split unit according to the intended deployment. It may be a whole video, camera, location, recording day, or individual subject. Keep related samples together when their similarity could leak information. Then use validation data for model selection and reserve the test set for the final assessment. Repeatedly tuning against the test set turns it into another development set.
Coverage matters as much as count. Include the lighting, object sizes, viewpoints, and difficult conditions that the application is expected to handle. Review labels and use duplicate checks as supporting tools, not as proof of independence. Two images from the same scene can be highly related without being exact duplicates. Document the split rule and any missing conditions so that performance claims match the evidence.
A one-minute clip produces one,eight hundred frames at thirty frames per second. Randomly assigning frames to train and test can place near-identical moments in both groups. Splitting by complete recording sessions gives a more demanding test of performance on a new session.