Make Vision Fast Enough for the Workflow
A model may predict quickly while the user still waits several seconds for a box.
Make Vision Fast Enough for the Workflow explained visually
A model may predict quickly while the user still waits several seconds for a box. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A model may predict quickly while the user still waits several seconds for a box.
Measure the complete path
A model may predict quickly while the user still waits several seconds for a box. End-to-end latency includes loading the image, decoding it, preparing the input, running inference, processing the output, and transferring the result. Measuring only the neural network can hide the real bottleneck.
Distinguish two speed questions
Latency is the time required for one request to complete. Throughput is the number of requests processed over a period. Batching can improve throughput while making one user wait longer. Model reuse can avoid repeated startup costs, and smaller inputs can reduce computation, but small objects may become harder to detect. Every speed change should be evaluated together with the accuracy needed by the task.
Optimize with evidence
Measure a cold first request and repeated warm requests separately. Also test realistic image sizes and concurrent users. A cache must be keyed correctly so that one project's model or image is never confused with another. Bound memory use and provide a clear failure path. For interactive annotation, smooth editing and predictable response times may matter more than maximizing the number of images processed per second.
A worked example
Suppose image loading takes zero point eight seconds, preprocessing zero point one seconds, inference zero point two seconds, and the remaining work zero point one seconds. Total latency is one point two seconds. Cutting inference time in half saves only zero point one seconds; improving loading may offer a larger benefit.
Worked example
Suppose image loading takes 0.8 seconds, preprocessing 0.1 seconds, inference 0.2 seconds, and the remaining work 0.1 seconds. Total latency is 1.2 seconds. Cutting inference time in half saves only 0.1 seconds; improving loading may offer a larger benefit.
Quick check: test your understanding
Try it yourself: List the stages you would time when a tracking request feels slow. Which measurements would you take before changing the model?
Check the reasoning
Measure storage access, decoding, model loading, preprocessing, inference, postprocessing, and response transfer, including cold and warm runs.
Does higher throughput always mean faster response for one user?
No. Queueing and batching can raise throughput while increasing individual waiting time.
How would you solve the practice case?
Measure storage access, decoding, model loading, preprocessing, inference, postprocessing, and response transfer, including cold and warm runs.
What does the worked example show?
Suppose image loading takes 0.8 seconds, preprocessing 0.1 seconds, inference 0.2 seconds, and the remaining work 0.1 seconds. Total latency is 1.2 seconds. Cutting inference time in half saves only 0.1 seconds; improving loading may offer a larger benefit.
Further reading
Programming Computer Vision with Python — Jan Erik Solem.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 13 transcript227 words · Video transcript · Matches the published narration and captionsOpen transcript
A model may predict quickly while the user still waits several seconds for a box. End-to-end latency includes loading the image, decoding it, preparing the input, running inference, processing the output, and transferring the result. Measuring only the neural network can hide the real bottleneck.
Latency is the time required for one request to complete. Throughput is the number of requests processed over a period. Batching can improve throughput while making one user wait longer. Model reuse can avoid repeated startup costs, and smaller inputs can reduce computation, but small objects may become harder to detect. Every speed change should be evaluated together with the accuracy needed by the task.
Measure a cold first request and repeated warm requests separately. Also test realistic image sizes and concurrent users. A cache must be keyed correctly so that one project's model or image is never confused with another. Bound memory use and provide a clear failure path. For interactive annotation, smooth editing and predictable response times may matter more than maximizing the number of images processed per second.
Suppose image loading takes zero point eight seconds, preprocessing zero point one seconds, inference zero point two seconds, and the remaining work zero point one seconds. Total latency is one point two seconds. Cutting inference time in half saves only zero point one seconds; improving loading may offer a larger benefit.