Optimizers

Why it matters

The update rule decides how fast and how reliably training finds good weights. Plain gradient descent works but is slow in the landscapes neural networks actually have. Momentum and Adam fix most of that, and AdamW is the optimizer behind nearly every LLM you've used. Knowing what each one does tells you which knobs to turn when training misbehaves.

What you'll be able to do

• Explain why plain SGD zig-zags in long, narrow valleys
• Describe momentum and per-parameter step sizes, and how Adam combines them
• Compare SGD, momentum and Adam on a real training run
• Know why AdamW is the default for training transformers, and what it costs in memory

Before this lesson

• Mini-batches and SGD

Next in this phase: Normalization layers