Glossary · Phase 6 · Modern LLM engineering
DPO
Direct preference optimization: training straight from preference pairs, without a separate reward model.
Learn it properly
DPO and direct preference methods · Preference training without a separate reward model. About 40 minutes.
Other terms from the same lesson
- implicit reward: In DPO, the score you can read off from how much more likely the trained model makes an answer than the reference model does.
- preference pair: One prompt with two answers, one marked as preferred (chosen) and one not (rejected).
- reference model: A frozen copy of the model before preference training. Training is penalized for drifting too far from it.