Phase 2 · Math, when you need it · Probability & statistics
Samples and confidence
Estimate from a sample and say how sure you are.
Open this lesson Create a free account
Why it matters
Every number from data is an estimate: a conversion rate from last week's visitors, an eval score from 50 test questions, a satisfaction score from 200 survey replies. Run it again on a different sample and you'd get a different number. A confidence interval says how different, which turns "the new prompt scored 84% vs 80%" into either "that's a real improvement" or "that's within the noise of 50 questions". It's the most useful statistics habit for anyone shipping AI features.
What you'll be able to do
- Explain why an estimate from a sample varies, and how much (the standard error)
- Compute a 95% confidence interval for a mean and for a rate
- Use the bootstrap to get an interval for any statistic
- Work out how many observations you need for a given precision
Before this lesson
Next in this phase: A/B tests and significance