Glossary · Phase 6 · Modern LLM engineering
Reference model
A frozen copy of the model before preference training. Training is penalized for drifting too far from it.
Learn it properly
DPO and direct preference methods · Preference training without a separate reward model. About 40 minutes.
Other terms from the same lesson
- DPO: Direct preference optimization: training straight from preference pairs, without a separate reward model.
- implicit reward: In DPO, the score you can read off from how much more likely the trained model makes an answer than the reference model does.
- preference pair: One prompt with two answers, one marked as preferred (chosen) and one not (rejected).