Language modeling

Why it matters

Every LLM, from GPT-2 to Claude, is trained on one deceptively simple task: given the text so far, predict the next token. Everything else (grammar, facts, style, even reasoning) is learned because it helps with that prediction. This phase builds a language model from counting to a working transformer, and it starts by making "predict the next token" and "how good is the model" precise.

What you'll be able to do

• Define a language model as a probability distribution over the next token
• Break a text's probability into next-token predictions with the chain rule
• Score a model with average cross-entropy (loss) and perplexity
• Build and evaluate the simplest baselines: uniform and unigram

Before this lesson

• The training loop
• Cross-entropy and KL divergence

Next in this phase: Tokenization and BPE