Phase 6 · Modern LLM engineering · Post-training
RLHF
Reward models and reinforcement learning from human feedback.
Open this lesson Create a free account
Why it matters
ChatGPT and GPT-3 started from the same kind of pretrained model. What made ChatGPT feel so different was a training step called reinforcement learning from human feedback, or RLHF. It trains the model to write the answers people prefer. Its parts now show up all over post-training: a learned score, a model pushed to raise that score, and a fine for drifting too far from where it started. Its main weakness is a model that learns to game a flawed score. That problem is one of the central puzzles in AI safety.
What you'll be able to do
- Train a reward model on preference pairs with the Bradley-Terry loss
- Use a reward model to pick the best of several sampled answers
- Explain the KL-regularized objective that RLHF optimizes, and what β controls
- Recognize reward hacking and why RLHF needs a leash on the policy
Inside this lesson
- 13 short sections
- 4 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 6 review cards for spaced repetition
What it covers
- The recipe
- Step 2: a reward model
- Worked example: the loss for four pairs
- Training a reward model on real preferences
- Step 3: reinforcement learning, on a leash
- Worked example: two answers, four leashes
- Watching reward hacking happen
- How the RL step actually runs
A taste of the code
import math
def bt_loss(r_chosen, r_rejected):
gap = r_chosen - r_rejected
chance = 1 / (1 + math.exp(-gap)) # sigmoid: the chance the chosen answer wins
return -math.log(chance)
for chosen, rejected in [(3.0, 1.0), (1.0, 1.0), (1.0, 3.0), (13.0, 11.0)]:
print(chosen, rejected, round(bt_loss(chosen, rejected), 3))Try a quiz question
Why does RLHF penalize the KL divergence from the SFT model?