Glossary · Phase 6 · Modern LLM engineering
Benchmark
A standard test set used to compare models. Useful, but models can be tuned to the test.
Learn it properly
Benchmarks and their limits · What leaderboards measure, and what they miss. About 30 minutes.
Other terms from the same lesson
- contamination: When a benchmark's test questions (or their answers) leaked into a model's training data, so its score overstates what it can really do.
- Goodhart's law: "When a measure becomes a target, it ceases to be a good measure": optimize for a score and the score stops meaning what it did.
- training cutoff: The date a model's training data stops. It knows nothing about events after it unless you give it the information.