Phase 2 · Math, when you need it · Calculus
Gradients
The vector of all partial derivatives: which way is uphill.
Open this lesson Create a free account
Why it matters
The gradient is the single object that training revolves around. For a model with a billion weights it's a vector of a billion numbers, one per weight, and together they say which direction in weight space makes the loss rise fastest. Step the other way and the loss falls. "Gradient descent", "vanishing gradients", "gradient clipping", "gradient accumulation": all of them are about this one vector.
What you'll be able to do
- Collect a function's partial derivatives into its gradient vector
- Explain why the gradient points uphill and the negative gradient downhill
- Derive and vectorize the gradient of mean squared error for a linear model
- Check an analytic gradient against a numerical one
Before this lesson
Next in this phase: Optimization by walking downhill