Glossary · Phase 6 · Modern LLM engineering
Continuous batching
Letting new requests join a running batch the moment a slot frees up, instead of waiting for the whole batch to finish. It keeps the GPU busy.
Learn it properly
Serving models · Batching, throughput and latency in production. About 35 minutes.
Other terms from the same lesson
- PagedAttention: Storing each conversation's KV cache in small fixed-size blocks, like pages of memory, so little memory is wasted and more conversations fit at once.
- throughput: How much work a system gets through per second, such as tokens generated across all users. Different from how fast one user gets a reply.