Glossary · Phase 4 · Deep learning
GELU
A smooth cousin of ReLU used in most transformers. It lets small negative values through a little instead of cutting them to 0.
Learn it properly
Activation functions · ReLU, sigmoid, GELU, and why nonlinearity matters. About 25 minutes.
Other terms from the same lesson
- activation function: The simple function a neuron applies after its weighted sum. Without it, stacking layers would add nothing.
- ReLU: The most common activation function: keep positive numbers, turn negative ones into 0.
- tanh: An S-shaped activation like sigmoid, but its outputs run from -1 to 1 instead of 0 to 1.