Batch Normalization
Stabilize internal layer values while preserving the scale and shift required by the task.
Batch Normalization explained visually
Follow mini-batch statistics as they center, scale, and stabilize hidden pre-activations.
What you will learn
Initialize, normalize, and stabilize signals and gradients in deeper networks.
Normalize a mini-batch
Batch normalization calculates a mean and variance for a neuron's pre-activations across examples.
Learn scale and shift
Trainable parameters allow the layer to recover the distribution required by the task.
Distinguish input normalization
Input normalization preprocesses raw features, whereas batch normalization operates inside the network.
Handle inference correctly
Training uses current batches and running estimates; inference uses stored running statistics.
Quick check: test your understanding
What values are normalized?
Layer pre-activations across a mini-batch.
Why learn scale and shift?
So the network can represent the distribution required by the task.
What happens during inference?
Stored running statistics are used.
Accessible referenceLesson 39 transcript357 words · Matches the published narration and captionsOpen transcript
Even with good initialization, the values moving through a network can change as the weights update. Batch normalization helps keep those internal signals within a manageable range. Imagine a factory with several processing stations. If one station suddenly sends parts that are ten times larger
than expected, every later station must constantly readjust. A more consistent input makes the entire production line easier to control. Inside a neural network, each neuron first calculates a weighted sum, sometimes called a pre-activation value. During training, batch normalization examines those values across the examples in
a mini-batch. Suppose a mini-batch contains sixteen examples. One neuron produces sixteen pre-activation values, one for each example. Batch normalization calculates their mean and variance. It then centers and scales the values so the batch has an approximate mean of zero and a standard deviation near one.
The layer does not remain permanently restricted to that exact distribution. It learns an additional scale and shift, allowing the network to represent the values needed for the task. Batch normalization can stabilize training, support larger learning rates, and reduce the chance that activations or gradients become extremely small or large.
It differs from input normalization. Input normalization preprocesses raw features before they enter the network. For example, image pixels may be scaled from zero through 255 into zero through one. Batch normalization operates inside the network. It normalizes the values produced by a layer across multiple examples
in the mini-batch. Training and inference also behave differently. During training, the layer uses statistics from the current mini-batch and maintains running estimates. During inference, it uses the stored running statistics so one prediction does not depend on which other examples happen to appear beside
it. Batch size can affect those estimates. Very small batches may produce noisy statistics, so other normalization methods can work better in some architectures. Batch normalization does not replace careful initialization or a suitable learning rate. It works with those choices to make optimization more stable.
We have now seen how initialization and normalization protect signal flow. In Lesson 40, we will examine what happens when gradients still vanish, explode, or leave ReLU neurons inactive.