Build a tiny GPT

Why it matters

This is the moment the phase has been building toward: a working GPT, written from parts you understand, trained on real text, generating new text. It's tiny (about 40,000 parameters, trained for a few hundred steps in your browser), but it's the same architecture and the same training objective as the models you use every day, and its mistakes are instructive.

What you'll be able to do

• Prepare a corpus for a language model: tokenize, split, and batch with shifted targets
• Train a small GPT from scratch and watch its loss fall
• Generate text from it and judge what it has and hasn't learned
• Connect what you see to how large language models behave

Before this lesson

• The GPT architecture
• Tokenization and BPE

Next in this phase: Encoders and embedding models