Evals

Why it matters

Every change to an LLM app (a reworded prompt, a new model, a cheaper model) can quietly fix three cases and break five. An eval is a fixed set of test inputs with known good answers and a script that scores your system against them. It's an A/B test for prompts: instead of "this feels better", you get "accuracy went from 0.80 to 0.90 and nothing else got worse". Teams that ship reliable AI features all have evals; it's the habit that matters most after prompting.

What you'll be able to do

• Explain what an eval is and why trying a few prompts by hand isn't enough
• Build a labeled dataset and choose a grader for it
• Compute accuracy, per-label recall and a majority baseline
• Use an eval to compare two prompts or two models before shipping a change

Before this lesson

• Structured output

Next in this phase: LLM-as-judge