Phase 6 · Modern LLM engineering · Inference
Serving models
Batching, throughput and latency in production.
Open this lesson Create a free account
Why it matters
Training a model is a one-time cost; serving it is forever. How a provider batches requests, manages GPU memory and routes traffic decides what you pay per token, how fast answers stream, and why prices drop for work that can wait. If you ever host an open model, these are the decisions you'll be making.
What you'll be able to do
- Explain why batching is essential for serving LLMs, and how continuous batching works
- Trade off throughput against per-user speed with a simple cost model
- Estimate how many conversations fit on a GPU, and why paged KV caches help
- Name the metrics and tools of production serving (TTFT, tokens/s, p95, vLLM)
Before this lesson
Next in this phase: Speculative decoding