Phase 5 · Transformers · Scale & inference
Scaling laws
How loss falls predictably with size, data and compute.
Open this lesson Create a free account
Why it matters
Scaling laws are why AI moved so fast: they told labs that bigger models trained on more data would get predictably better, which justified spending hundreds of millions of dollars on a single training run. They also explain decisions that shape the models you can buy, such as why small models trained on enormous datasets are so capable and cheap to run, and what may slow progress.
What you'll be able to do
- Explain what a scaling law is: loss falling as a power law in parameters, data and compute
- Use the Chinchilla result to split a compute budget between model size and training tokens
- Fit a power law to small runs and extrapolate it, the way labs plan large runs
- Explain why today's models are often trained far past the 'compute-optimal' point
Before this lesson
Next in this phase: Inference and the KV cache