Inference and the KV cache

Why it matters

Every time a model streams a reply, it's running the loop from the tiny-GPT lesson, one token at a time. Done naively, that loop gets slower with every token. The KV cache is the trick that makes generation practical, and it explains everyday facts about LLM APIs: why the first token takes longest, why long conversations cost more, why output tokens are priced higher than input tokens, and why cached prompts are cheaper and faster.

What you'll be able to do

• Describe the two phases of generation: prefill and decode
• Implement attention with a KV cache and confirm it matches full recomputation
• Calculate how much memory a KV cache needs, and how grouped-query attention shrinks it
• Explain time to first token, tokens per second, and why prompt caching is cheaper

Before this lesson

• The GPT architecture

Next in this phase: Long context