Glossary · Phase 4 · Deep learning
Pre-norm
Putting the normalization before each sub-layer instead of after it. It makes deep transformers much easier to train.
Learn it properly
Normalization layers · Batch and layer norm keep training stable. About 30 minutes.
Other terms from the same lesson
- batch normalization: Rescaling each feature across a batch to an average of 0 and a spread of 1, then letting the network adjust it. Common in image models.
- normalization: Rescaling the numbers inside a network so they stay in a steady range, which makes training smoother.
- RMSNorm: A simpler LayerNorm that only rescales by the typical size of the numbers, without subtracting the average. Used in many recent LLMs.