Glossary · Phase 6 · Modern LLM engineering
Speculative decoding
Speeding up generation by letting a small model draft several tokens and a big model check them at once.
Learn it properly
Speculative decoding · Draft with a small model, verify with a big one. About 35 minutes.
Other terms from the same lesson
- draft model: In speculative decoding, the small fast model that guesses the next few tokens for the big model to check.
- lossless: Changing how something is computed without changing the result at all, such as speculative decoding producing exactly what the big model would have.