Phase 6 · Modern LLM engineering · Post-training
DPO and direct preference methods
Preference training without a separate reward model.
Open this lesson Create a free account
Why it matters
RLHF works but is expensive and fiddly: a reward model, sampling during training, four models in memory. Direct Preference Optimization (2023) gets much of the benefit with a loss you can write in one line and train like SFT. It quickly became a standard step in post-training open models, and it's the method you're most likely to use if you ever tune a model on your own preferences.
What you'll be able to do
- Explain how DPO turns preference pairs into a loss on the model itself, with no reward model
- Compute sequence log-probabilities and the DPO loss for chosen and rejected answers
- Run DPO on a real model and evaluate what changed, including side effects
- Know the main variants and when DPO is preferred over RLHF with PPO
Before this lesson
Next in this phase: Constitutional AI