PPO

Also written: proximal policy optimization

Proximal policy optimization: the reinforcement-learning method classic RLHF uses, which nudges the model toward higher-scoring answers in small, careful steps.

Other terms from the same lesson

• best-of-n sampling: Generating several answers and keeping the one a reward model scores highest. Simple and strong, but it costs n generations per question.
• policy: In reinforcement learning, the model being trained: the thing that decides what to do (here, which token to write next).
• reward hacking: When a model finds ways to score highly with the reward model that don't make its answers better, like padding or flattery.
• reward model: A model that scores answers by how much people would prefer them.
• RLHF: Reinforcement learning from human feedback: training a model towards answers people prefer, using a learned scorer.
• value model: In RLHF with PPO, a helper model that predicts the score an answer is likely to get from the part written so far.