Phase 6 · Modern LLM engineering · Post-training
Constitutional AI
Training with AI feedback guided by written principles.
Open this lesson Create a free account
Why it matters
Human preference labels are slow, expensive, inconsistent, and unpleasant to collect for harmful content. Constitutional AI, introduced by Anthropic in 2022, replaces most of them with feedback from a model that follows a written list of principles. The idea now runs through post-training at most labs, and it changes where the important decisions live: in the text of the principles.
What you'll be able to do
- Describe Constitutional AI's two phases: self-critique and revision, then RL from AI feedback
- Show how the written principles decide which answers count as better
- Measure over-refusal and missed harms, and explain why keyword rules fail at both
- Recognize where AI feedback is used today and its limits
Before this lesson
Next in this phase: Training reasoning models