Every eval run gives a slightly different number. The job is to tell a real change from a wobble.
Why a tester cares
A statistical test asks how often two runs would differ this much if nothing had changed. Get it wrong and you either ship a regression or chase noise for a week.
Words you will meet
Regression:
something that used to work and now works worse.
p-value:
the chance of seeing a difference this big by luck alone.
Significant:
unlikely to be luck, usually meaning a p-value below 5%.
Every eval run produces a number, and every number is a little different. The job is to tell a real change from noise, so the team neither ships a regression nor wastes a week chasing a wobble.
For two independent runs, a two-proportion test turns "80% then 74%" into a probability: if nothing had really changed, how often would runs differ this much? Below 5%, call it real. Two things decide the answer: how big the change is, and how many cases each run had. The same 8-point drop is noise on 100 cases and very likely real on 500.
Your task
Label each of the twelve before-and-after pairs real or noise. Use the calculator, or Python on your own machine, to test each one at the 5% level.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
1P01: before 180/200 (90.0%), after 176/200 (88.0%)
2P02: before 180/200 (90.0%), after 160/200 (80.0%)
3P03: before 150/200 (75.0%), after 158/200 (79.0%)
4P04: before 190/200 (95.0%), after 182/200 (91.0%)
5P05: before 170/200 (85.0%), after 150/200 (75.0%)
6P06: before 120/200 (60.0%), after 130/200 (65.0%)
7P07: before 195/200 (97.5%), after 185/200 (92.5%)
8P08: before 100/200 (50.0%), after 112/200 (56.0%)
9P09: before 160/200 (80.0%), after 156/200 (78.0%)
10P10: before 175/200 (87.5%), after 140/200 (70.0%)
11P11: before 60/100 (60.0%), after 52/100 (52.0%)
12P12: before 400/500 (80.0%), after 370/500 (74.0%)
How this is checked: Each row has a right label. At least 9 of 12 must match. For the full score, a harder check: every real one must be caught, with at most 1 false alarm.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Two proportion z test for eval results: telling a real regression from noise
AI Testing for QA Engineers·Intermediate·Statistics for testers
How it works
Every eval run produces a slightly different number, and the job is to tell a real change from a wobble, so the team neither ships a regression nor spends a week chasing noise.
For two independent runs, a two-proportion test asks: if nothing had really changed, how often would two runs differ this much? When that probability is below the usual 5%, the change is called real. Two things decide the answer: the size of the change and how many cases each run had. The same drop can be noise on a small set and almost certainly real on a large one.
About the assignment
Label twelve before-and-after pairs as real changes or noise, using the calculator or your own Python.
What you will do
1Enter each pair's passes and totals into the calculator.
2Read the probability and compare it with 5%.
3Label the pair real only when the probability is below it.
Common mistakes
Judging by the size of the drop alone
A big drop on few cases can be noise, and a small drop on many cases can be real.
Using this test on the same cases
When both versions ran on the same cases, a paired test is stronger. Lesson 44 covers it.
In a real test suite
Teams usually automate this test in the gate script and print its result next to the two pass rates, so nobody has to argue about whether a drop matters.
The 5% line is a convention, not a law. For checks that guard money or safety, many teams use a stricter line, or treat any confirmed failure as blocking.
