Phase 5 · Transformers · Scale & inference
Pretraining at scale
Data, compute and what models learn from the internet.
Open this lesson Create a free account
Why it matters
Your tiny GPT trained on 9,000 tokens in seconds. A frontier model trains on tens of trillions of tokens for months on thousands of GPUs, with the same objective. Knowing what goes into that run (which data, how much compute, what it costs) explains a lot about the models you use: what they know, why they have a cutoff date, why they cost what they do, and why only a few companies train them.
What you'll be able to do
- Describe the pretraining pipeline: collect, filter, deduplicate, mix and tokenize text
- Estimate training compute with the 6·N·D rule and turn it into GPU time
- Explain what a pretrained (base) model knows, and where its knowledge cutoff comes from
Before this lesson
Next in this phase: Scaling laws