Xavier and He Initialization
Match the initial weight scale to layer connectivity and activation behavior for healthier signal flow.
Xavier and He Initialization explained visually
Compare fading and growing signals to understand when Xavier or He initialization is appropriate.
What you will learn
Initialize, normalize, and stabilize signals and gradients in deeper networks.
Preserve signal strength
Initialization should keep activations and gradients from shrinking or growing uncontrollably across many layers.
Use Xavier with smooth activations
Xavier considers input and output connections and commonly suits sigmoid or tanh networks.
Use He with ReLU
He initialization uses a larger variance to compensate for ReLU units that block negative signals.
Inspect signal distributions
Activation and gradient distributions reveal whether the selected scale suits the architecture.
Quick check: test your understanding
What does Xavier initialization consider?
The numbers of input and output connections.
Which method commonly suits ReLU?
He initialization.
Does initialization guarantee success?
No. Data, architecture, optimization, and learning rate still matter.
Accessible referenceLesson 38 transcript350 words · Matches the published narration and captionsOpen transcript
Random initialization breaks symmetry, but the scale of the random values still matters. Imagine passing a message through a long line of people. If every person whispers more quietly than the person before, the message disappears. If every person shouts louder, the message becomes distorted.
A deep network faces a similar problem as activations and gradients move through many layers. Xavier and He initialization choose the weight scale based on the number of connections in a layer. Their goal is to keep signals from shrinking toward zero or growing uncontrollably.
Xavier initialization considers the number of inputs and outputs connected to the layer. It draws random weights with a variance that decreases as those connection counts increase. This scaling helps preserve a balanced signal through forward and backward propagation. Xavier initialization commonly works well with activation functions such as sigmoid
and hyperbolic tangent, especially when their inputs remain near the sensitive central region instead of saturating at the extremes. He initialization uses a larger variance based mainly on the number of input connections. It was designed for ReLU and related activation functions. ReLU changes every negative input to zero.
As a result, some neurons do not pass a signal for a given example. He initialization compensates with a scale that helps the remaining active neurons maintain useful activation strength. The choice should therefore match the activation function. Xavier provides a common starting point for sigmoid or tanh networks.
He initialization usually provides a better starting point for ReLU networks. Neither method guarantees successful training. Data quality, architecture, learning rate, and optimization still matter. Initialization simply gives the optimizer a healthier place to begin. During an experiment, we can compare the distributions of activations and
gradient magnitudes across layers. If they rapidly collapse toward zero or grow dramatically, the initialization may not fit the architecture. Proper initialization reduces the chance of unstable early training and often helps the network converge faster. It also supports deeper models by keeping information at a useful scale. Initialization controls the starting weights. Another technique stabilizes the activations while training continues. Lesson 39 introduces batch normalization.