Phase 4 · Deep learning · Training
Vanishing and exploding gradients
Why deep networks were hard to train.
Open this lesson Create a free account
Why it matters
For decades, networks deeper than a few layers simply didn't train, and nobody could fully say why. The answer is in the chain rule: backprop multiplies one factor per layer, and a long product of numbers slightly below or above 1 heads to zero or infinity. Every major trick in this phase (ReLU, careful initialization, normalization, residual connections) exists to keep that product near 1, and the same problem shapes how LLMs are built.
What you'll be able to do
- Explain why gradients shrink or grow exponentially with depth
- Measure gradient size layer by layer in a real network
- Connect the fixes: activations, initialization, normalization, residual connections
- Use gradient clipping against exploding gradients
Before this lesson
Next in this phase: Debugging training