Compute-optimal

The split of a fixed compute budget between model size and training tokens that gives the lowest loss: about 20 tokens per parameter.

Other terms from the same lesson

• irreducible loss: The floor the loss approaches however big the model and data get, because some text is simply unpredictable.
• log-log plot: A chart where both axes use a log scale, so each step along an axis multiplies by 10. Power laws turn into straight lines.
• overtraining: Training a model on far more tokens than compute-optimal. Training costs more, but the smaller model is cheaper to run for every user afterwards.
• parameter count: How many adjustable numbers (weights) a model has. Large language models have billions.
• power law: A curve where each doubling of the input cuts the output by the same fraction. It shows up as a straight line on a log-log plot.
• scaling laws: The finding that models improve predictably as you add data, size and compute.