Phase 5 · Transformers · Scale & inference
Long context
Efficient attention and million-token windows.
Open this lesson Create a free account
Why it matters
Context windows grew from about 2,000 tokens in 2020 to hundreds of thousands, and in some models a million or more, which is enough for whole codebases, contracts or years of campaign reports in one prompt. That changed how AI applications are built. Knowing what makes long context hard tells you what it costs, where models still struggle with it, and when retrieval (RAG) is still the better choice.
What you'll be able to do
- Explain why attention's cost grows with the square of the context length
- Implement tiled attention with a running softmax, the idea behind FlashAttention
- Describe sliding-window attention and how models are extended to longer contexts
- Judge when a long context is the right tool and how to lay out a long prompt
Before this lesson
Next in this phase: Mixture of experts