Phase 6 · Modern LLM engineering · Systems & safety
Evaluating agents
Measure multi-step, tool-using behavior.
Open this lesson Create a free account
Why it matters
An agent that answers correctly 9 times out of 10 sounds good until you notice that, across a day of real conversations, the tenth time it refunds the wrong order. Agents act, so evaluating them means checking what they did, how consistently, and what it cost. This is the evaluation discipline from Phase 1 extended to multi-step, tool-using systems, and it decides whether an agent is ready to ship.
What you'll be able to do
- Evaluate agents by the end state they produce, not just their final message
- Measure consistency with pass^k and contrast it with pass@k
- Track safety and efficiency alongside success: wrong actions, tool calls, cost
- Know the main agent benchmarks and how to build a test environment for your own agent
Before this lesson
Next in this phase: Reading AI papers