Phase 4 · Deep learning · Neural networks
Activation functions
ReLU, sigmoid, GELU, and why nonlinearity matters.
Open this lesson Create a free account
Why it matters
The activation is the small bend that gives a neural network its power. Without it, a hundred layers are no more capable than one. Which bend you use decides how easily the network trains: the switch from sigmoid to ReLU was one of the changes that made deep learning work, and the GELU and SwiGLU used inside today's LLMs are the latest step in the same story.
What you'll be able to do
- Show that stacked layers without activations collapse into one linear layer
- Compare sigmoid, tanh, ReLU, leaky ReLU and GELU, and their slopes
- Explain saturation and dead ReLUs, and why they slow or stop learning
- Choose the right activation for hidden layers and for outputs
Before this lesson
Next in this phase: Multi-layer networks