Phase 4 · Deep learning · Architectures
Residual connections
Skip connections that let gradients flow.
Open this lesson Create a free account
Why it matters
In 2015, ResNets trained networks with 150 layers by making one tiny change: each block adds its output to its input instead of replacing it. That skip connection is now everywhere. Every transformer block in every LLM is wrapped in two of them, and the "residual stream" they create is how researchers describe what flows through a language model.
What you'll be able to do
- Write a residual block, y = x + f(x), and explain the identity path
- Show why it keeps gradients flowing through very deep stacks
- See why residual connections need normalization or scaled initialization
- Read the transformer's 'residual stream' in these terms
Before this lesson
Next in this phase: Convolutional networks