Why Weight Initialization Matters
Break neuron symmetry and begin training with diverse signals at a useful numerical scale.
Why Weight Initialization Matters explained visually
Use students exploring a maze to see why identical starts preserve symmetry and random starts create diversity.
What you will learn
Initialize, normalize, and stabilize signals and gradients in deeper networks.
Avoid identical neurons
Zero weights can make neurons in the same layer produce identical outputs and receive identical gradients.
Break symmetry
Small random differences allow neurons to follow different learning paths and discover different features.
Choose a useful scale
Very large initial values can amplify signals, while very small values can make them disappear.
Make comparisons reproducible
A fixed random seed lets model designs begin from the same parameter values.
Quick check: test your understanding
What is the symmetry problem?
Neurons remain identical instead of learning different features.
Why initialize randomly?
To give neurons different starting points.
Why fix a random seed?
To compare experiments from the same starting conditions.
Accessible referenceLesson 37 transcript348 words · Matches the published narration and captionsOpen transcript
Training adjusts a neural network's weights, but every weight needs a starting value. This starting point can strongly influence how well the network learns. Imagine several students entering a dark maze. If they all begin in exactly the same place, face the same direction, and follow the same rule, they will explore identical
paths. Adding more students would not help because none of them would discover anything different. The same problem appears when every weight in a neural-network layer starts at zero. Neurons in that layer receive the same inputs, produce the same outputs, and receive identical gradients.
After each update, their weights remain identical. The neurons continue behaving like copies instead of learning different features. This is called the symmetry problem. For an image classifier, one neuron might need to respond to vertical edges while another responds to curves or textures. Identical weights prevent that diversity from developing.
Random initialization breaks the symmetry. Each neuron begins with slightly different weights, so it receives different signals and follows a different learning path. However, random does not mean that any range will work well. Very large starting weights can make activations and gradients grow excessively.
Very small weights can cause signals and gradients to shrink as they pass through many layers. A simple method draws small values from a uniform distribution or a normal distribution centered near zero. This gives neurons different starting points, but a fixed random scale may still behave poorly when layers have very different numbers of
connections. Initialization also affects reproducibility. Two training runs with different random starting weights may follow different paths and finish with different results. During experiments, researchers often fix a random seed so comparisons between model designs remain fair. The goal of initialization is not to begin with accurate predictions.
It is to create diverse, well-scaled signals that allow useful gradients to move through the network from the first training step. Modern initialization methods choose the weight scale using the number of connections in each layer. In Lesson 38, we will compare two widely used methods: Xavier initialization and He initialization.