Inference

Using a trained model to make predictions, as opposed to training it.

Other terms from the same lesson

• grouped-query attention: Letting groups of query heads share one key head and one value head, so the KV cache shrinks with little loss in quality.
• KV cache: Saving the keys and values of earlier tokens so the model doesn't recompute them for every new token.
• memory bandwidth: How many bytes per second a chip can read from its memory. Generating text one token at a time is usually limited by this, not by arithmetic.
• multi-query attention: The extreme of grouped-query attention: every query head shares a single key head and value head, giving the smallest KV cache.