Phase 5 · Transformers · The transformer
Encoders and embedding models
BERT-style models and where they fit.
Open this lesson Create a free account
Why it matters
GPT-style decoders get the headlines, but encoders quietly power much of applied AI: the embedding models behind every RAG system in Phase 1, rerankers that sort search results, classifiers that tag millions of support tickets cheaply. They're the same transformer blocks with one change, attention that looks both ways, and a different training game.
What you'll be able to do
- Contrast encoders (bidirectional attention) with decoders (causal attention)
- Explain masked language modeling, the training objective behind BERT
- Describe how encoders produce embeddings, classifications and reranking scores
- Choose between encoder, decoder and encoder-decoder models for a task
Before this lesson
Next in this phase: Pretraining at scale