Glossary · Phase 5 · Transformers
Log-log plot
A chart where both axes use a log scale, so each step along an axis multiplies by 10. Power laws turn into straight lines.
Learn it properly
Scaling laws · How loss falls predictably with size, data and compute. About 35 minutes.
Other terms from the same lesson
- compute-optimal: The split of a fixed compute budget between model size and training tokens that gives the lowest loss: about 20 tokens per parameter.
- irreducible loss: The floor the loss approaches however big the model and data get, because some text is simply unpredictable.
- overtraining: Training a model on far more tokens than compute-optimal. Training costs more, but the smaller model is cheaper to run for every user afterwards.
- parameter count: How many adjustable numbers (weights) a model has. Large language models have billions.
- power law: A curve where each doubling of the input cuts the output by the same fraction. It shows up as a straight line on a log-log plot.
- scaling laws: The finding that models improve predictably as you add data, size and compute.