Normalization layers

Why it matters

As a network trains, the scale of each layer's inputs keeps shifting, and in a deep network small shifts compound. Normalization layers re-center and re-scale activations inside the network so every layer sees well-behaved numbers. Batch norm made very deep image models practical; layer norm (and its cousin RMSNorm) sits in every block of every transformer, including the LLMs you use.

What you'll be able to do

• Explain what batch normalization and layer normalization compute, and over which axis
• Show a deep network that won't train without normalization and does with it
• Know batch norm's train/eval difference and why transformers use layer norm instead
• Recognize RMSNorm and pre-norm placement in modern LLMs

Before this lesson

• Weight initialization
• The training loop

Next in this phase: Dropout