Phase 1 · Building with LLMs · Quality & production
LLM-as-judge
Grade open-ended output with rubrics and a second model.
Open this lesson Create a free account
Why it matters
Code can check that a reply is valid JSON or under 60 characters. It can't check whether a reply is on-brand, answers the customer's actual question, or sticks to the policy it quotes. For those you can ask a second model to grade the first, against a rubric. Done carelessly, a judge adds confident noise. Done well, it lets you evaluate open-ended output (ad copy, support replies, summaries) on hundreds of examples in minutes.
What you'll be able to do
- Decide which quality checks need a model judge and which code can do
- Write a judge prompt with specific pass/fail criteria and structured output
- Compare two outputs fairly by judging in both orders
- Measure how often the judge agrees with human labels before trusting it
Before this lesson
Next in this phase: Cost and latency