Phase 5 · Transformers · Language modeling
Tokenization and BPE
How tokenizers are built from data.
Open this lesson Create a free account
Why it matters
In Phase 1 you counted tokens to estimate cost and context. Here's where tokens come from. A tokenizer is trained on data before the model is, and its choices ripple through everything: how much a prompt costs, how much fits in context, why models stumble on spelling or counting letters, and why some languages and formats are more expensive than others.
What you'll be able to do
- Explain why models use subword tokens rather than characters or whole words
- Run byte-pair encoding: repeatedly merge the most frequent adjacent pair
- See how a tokenizer's training data decides what's cheap and what's expensive to encode
- Connect tokenizer design to cost, context limits and quirks of LLM behavior
Before this lesson
Next in this phase: A bigram model