Optimization by walking downhill

Why it matters

Every neural network you've heard of was trained by the same loop: compute the loss, compute its gradient, take a small step against it, repeat. A large language model repeats that loop hundreds of thousands of times over trillions of tokens. In this lesson you'll run the loop yourself on a small problem and see exactly how the learning rate, the starting point and the shape of the loss decide whether training works.

What you'll be able to do

• Write the gradient descent update and explain each symbol
• Choose a learning rate by recognizing slow, good and diverging runs
• Fit a linear model to campaign data by gradient descent, from scratch
• Explain local minima, stochastic (mini-batch) gradients, and why optimizers like Adam exist

Before this lesson

• Gradients

Next in this phase: Probability basics