Glossary · Phase 6 · Modern LLM engineering
Throughput
How much work a system gets through per second, such as tokens generated across all users. Different from how fast one user gets a reply.
Learn it properly
Serving models · Batching, throughput and latency in production. About 35 minutes.
Other terms from the same lesson
- continuous batching: Letting new requests join a running batch the moment a slot frees up, instead of waiting for the whole batch to finish. It keeps the GPU busy.
- PagedAttention: Storing each conversation's KV cache in small fixed-size blocks, like pages of memory, so little memory is wasted and more conversations fit at once.