Phase 4 · Deep learning · Training
Mini-batches and SGD
Train on small random batches instead of everything at once.
Open this lesson Create a free account
Why it matters
No one computes the gradient over a whole dataset before taking a step: LLMs train on trillions of tokens. Instead each step uses a small random batch, which gives a noisy but cheap estimate of the true gradient. That's stochastic gradient descent, and the batch size you pick affects speed, memory, stability and even how well the model generalizes.
What you'll be able to do
- Explain full-batch, stochastic and mini-batch gradient descent, and epochs vs steps
- Train with a shuffled DataLoader
- Describe how batch size trades speed, noise and memory
- Use gradient accumulation to imitate a larger batch
Before this lesson
Next in this phase: Optimizers