Glossary · Phase 5 · Transformers
Grouped-query attention
Letting groups of query heads share one key head and one value head, so the KV cache shrinks with little loss in quality.
Learn it properly
Inference and the KV cache · Why generation is fast after the first token. About 35 minutes.
Other terms from the same lesson
- inference: Using a trained model to make predictions, as opposed to training it.
- KV cache: Saving the keys and values of earlier tokens so the model doesn't recompute them for every new token.
- memory bandwidth: How many bytes per second a chip can read from its memory. Generating text one token at a time is usually limited by this, not by arithmetic.
- multi-query attention: The extreme of grouped-query attention: every query head shares a single key head and value head, giving the smallest KV cache.