Glossary · Phase 5 · Transformers
Decoder
A transformer that reads left to right and predicts the next token; each token can only see the ones before it. GPT-style chat models are decoders.
Learn it properly
The transformer block · Attention, MLP, residuals and norms together. About 35 minutes.
Other terms from the same lesson
- feed-forward network: The small neural network inside each transformer block. It works on each token on its own, after attention has mixed in information from the other tokens.
- model card: The fact sheet published with a model: its size, what it was trained on, what it's for and its known limits.
- SwiGLU: A feed-forward layer design used in Llama-style models, where one set of numbers acts as a gate that turns another set up or down.
- transformer: The design behind today's language models: layers of attention plus small neural networks, repeated many times.
- transformer block: One repeating unit of a transformer: attention, then a small neural network, each with a shortcut and normalization.