Phase 6 · Modern LLM engineering · Inference
Speculative decoding
Draft with a small model, verify with a big one.
Open this lesson Create a free account
Why it matters
Speculative decoding makes large models generate two to three times faster with no change at all to what they say. Providers use it widely, and it's a neat example of a recurring idea in AI systems: let a cheap model do most of the work and an expensive one check it.
What you'll be able to do
- Explain why checking several tokens costs about the same as generating one
- Implement draft-and-verify decoding and confirm it gives exactly the large model's output
- Show that the accept/reject rule preserves the large model's distribution when sampling
- Judge what makes a good draft model and when speculation pays off
Before this lesson
Next in this phase: Running open models locally