Phase 2 · Math, when you need it · Probability & statistics
A/B tests and significance
Tell a real lift from noise.
Open this lesson Create a free account
Why it matters
You've probably run A/B tests through a tool that announces a winner. Underneath is a hypothesis test, and the same machinery decides whether a new prompt, model or retrieval setup really beats the old one on your eval. Knowing what the p-value does and doesn't mean, and how much data a test needs, protects you from shipping noise, and from calling a winner on day two.
What you'll be able to do
- State a null hypothesis and read a p-value correctly
- Run a two-proportion z-test and a confidence interval for the difference
- Plan a test's sample size from the baseline rate and the lift you care about
- Avoid the classic mistakes: peeking, many comparisons, and confusing significant with important
Before this lesson
Next in this phase: Maximum likelihood