Phase 5 · Transformers · Language modeling
A bigram model
The simplest language model, built from counts.
Open this lesson Create a free account
Why it matters
The bigram model is the simplest language model that uses context: it predicts each character from the one before it. It's small enough to see completely (one table), and it's the bridge to neural language models: train a single neural layer on the same task and it learns the very same table. Every later model in this phase is a better way of filling in that "what comes next" table.
What you'll be able to do
- Build a bigram language model from a table of next-character counts
- Generate text by sampling one character at a time
- Handle unseen pairs with smoothing, and measure the improvement over unigrams
- Train the same model as a one-layer neural network and see it reach the same answer
Before this lesson
Next in this phase: Sampling strategies