Glossary · Phase 5 · Transformers
Memory bandwidth
How many bytes per second a chip can read from its memory. Generating text one token at a time is usually limited by this, not by arithmetic.
Learn it properly
Inference and the KV cache · Why generation is fast after the first token. About 35 minutes.
Other terms from the same lesson
- grouped-query attention: Letting groups of query heads share one key head and one value head, so the KV cache shrinks with little loss in quality.
- inference: Using a trained model to make predictions, as opposed to training it.
- KV cache: Saving the keys and values of earlier tokens so the model doesn't recompute them for every new token.
- multi-query attention: The extreme of grouped-query attention: every query head shares a single key head and value head, giving the smallest KV cache.