The transformer block

Why it matters

A transformer is the same block repeated: 12 times in GPT-2 small, around 80 or more in the largest open models. Everything in this phase and the last comes together here: attention, the MLP, residual connections, layer normalization. Once you can write one block, a GPT is mostly a loop over blocks, which is the next lesson.

What you'll be able to do

• Assemble a block from attention, an MLP, residual connections and layer norms
• Explain the division of labor: attention moves information between tokens, the MLP processes each token
• Count a block's parameters (about 12·d² for width d)
• Verify the block keeps shapes and stays causal

Before this lesson

• Multi-head attention
• Residual connections
• Normalization layers

Next in this phase: The GPT architecture