Phase 5 · Transformers · The transformer
The GPT architecture
A stack of decoder blocks that predicts the next token.
Open this lesson Create a free account
Why it matters
GPT stands for Generative Pre-trained Transformer, and the architecture is strikingly simple: embed the tokens, run them through a stack of identical blocks, and turn the final vectors into next-token scores. GPT-2, GPT-3, Llama and the models behind today's assistants all share this skeleton, differing mainly in size, data and the details you've seen. After this lesson, a model card's numbers will mean something concrete.
What you'll be able to do
- Assemble a GPT: embeddings, a stack of blocks, a final norm and a language-model head
- Count a GPT's parameters and reproduce GPT-2 small's 124 million
- Write the generation loop: predict, sample, append, repeat
- Read a model card's configuration (layers, heads, width, context, vocabulary)
Before this lesson
Next in this phase: Build a tiny GPT