Glossary · Phase 5 · Transformers
Unigram
The simplest language model: it predicts each character or word from how common it is overall, ignoring what came before.
Learn it properly
Language modeling · Predict the next token, and score it with cross-entropy. About 30 minutes.
Other terms from the same lesson
- add-one smoothing: Adding 1 to every count before turning counts into chances, so nothing the model never saw gets a chance of exactly 0.
- corpus: The collection of text a language model learns from, such as product descriptions or a slice of the web.
- language model: A model that predicts the next token from the ones before it.