Deployment: Package the Whole Neural-Network System
Your network works in a notebook, but deployment introduces new ways to fail.
Deployment: Package the Whole Neural-Network System explained visually
Your network works in a notebook, but deployment introduces new ways to fail. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Your network works in a notebook, but deployment introduces new ways to fail.
Preserve the input contract
Your network works in a notebook, but deployment introduces new ways to fail. The application must load the right weights, apply the same preprocessing, interpret outputs correctly, and respond within its resource budget. Exporting a model is only one part of that system.
Measure real resource use
Package the architecture or runtime format, model version, class mapping, preprocessing rules, and required dependencies together. Test representative inputs in the actual runtime. Numerical differences can arise across formats, devices, or precision settings. Quantization and smaller input sizes may improve speed or memory use, but their accuracy effects must be measured on the task, especially for small or subtle objects.
Validate before activation
Separate training from interactive inference when the architecture requires different resources. Use queues to limit expensive work and avoid starting more jobs than the service can support. Keep cancellation and failure reporting responsive. For model updates, retain the previous successful artifact while the candidate is trained and validated. Activation should be a deliberate, testable step so a failed update does not remove a working model.
Worked example
A model expects RGB values normalized with a particular mean and standard deviation. A deployed endpoint supplies unnormalized BGR values instead. The tensor shape can be correct while predictions are wrong, which is why deployment tests must check input meaning as well as dimensions.
Quick check: test your understanding
Try it yourself: List the artifacts needed to reproduce a deployed image classifier beyond its weight file.
Check the reasoning
Include class names and order, preprocessing, model architecture or export format, dependency versions, inference settings, and a small known-input verification set.
Should a candidate model replace the active model before validation finishes?
No. Keep the prior successful model available until the candidate passes the required checks.
Further reading
Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.
Deep Learning — Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 80 transcript230 words · Video transcript · Matches the published narration and captionsOpen transcript
The model works in your notebook. You put it on the website, and suddenly the predictions look wrong. The weights may be fine. Something around them may have changed.
Deployment means packaging the whole prediction process. That includes the model version, input preparation, label mapping, and the runtime needed to execute it.
For example, suppose training used red, green, and blue channels with a particular normalization. The deployed code sends blue, green, and red channels without that normalization. The image dimensions still look correct, but the numbers now mean something different.
Test representative inputs through the actual deployed runtime. Compare outputs, measure response time, and check memory use. Faster settings, lower numerical precision, or smaller images may help performance, but they can also reduce accuracy. Small objects are one reason to inspect that trade-off carefully.
Training and interactive prediction may need different resources. Expensive training jobs can use a queue, while prediction needs a setup that responds within the application's requirements. Failure reporting and cancellation should remain responsive.
When updating the model, keep the previous successful version available. Train the candidate, validate it, and activate it only after it passes the required checks. Keep a way to roll back if production problems appear.
A deployed neural network is a system, not just a weights file. Reliable results depend on every step from incoming data to the prediction the user sees.