Normalization

Also written: layer norm, layer normalization, batch norm, LayerNorm

Rescaling the numbers inside a network so they stay in a steady range, which makes training smoother.

Other terms from the same lesson

• batch normalization: Rescaling each feature across a batch to an average of 0 and a spread of 1, then letting the network adjust it. Common in image models.
• pre-norm: Putting the normalization before each sub-layer instead of after it. It makes deep transformers much easier to train.
• RMSNorm: A simpler LayerNorm that only rescales by the typical size of the numbers, without subtracting the average. Used in many recent LLMs.