Weight initialization

Why it matters

Before the first training step, every weight needs a starting value. Get the scale wrong and a deep network's signals blow up to infinity or shrink to nothing within a few layers, and training stalls before it starts. Good initialization was one of the quiet fixes that made deep networks trainable, and it's still a line you'll see near the top of every LLM's code.

What you'll be able to do

• Explain why all-zero starting weights can't work (symmetry)
• Show how badly scaled weights make signals explode or vanish through many layers
• Choose Xavier/Glorot or He/Kaiming initialization from the activation function
• Recognize the initialization choices built into PyTorch layers and LLMs

Before this lesson

• Multi-layer networks

Next in this phase: The training loop