Skip to content
CodeKruPro

A pass rate is a range

41/99

Module · Statistics for testers

intermediate

about 35 min

What this lesson teaches

The idea

A pass rate from a small test set is an estimate, and an estimate needs a range. 18 out of 20 passing does not mean the real rate is exactly 90%.

Why a tester cares

The range shrinks as the number of cases grows, which is why bigger test sets give firmer claims. You can work it out in Python or with the calculator.

Words you will meet

Pass rate:
the share of test cases that passed.
Interval:
a range the true value probably lies in.
Wilson interval:
the usual formula for a pass-rate range that still works with few cases.

What you will do

  1. 1Pick Python or the calculator
  2. 2Compute the intervals
  3. 3Check the edge cases
  4. 4Submit
Jump to practice

"18 of 20 passed, so the bot is 90% accurate" is a sentence every AI team has said and should not. With 20 cases, the true pass rate could easily be anywhere from about 70% to 97%. A pass rate is an estimate, and an estimate needs a range.

The Wilson score interval is the usual choice for pass rates. It behaves well with small samples and at 0% or 100%, where the textbook "plus or minus" formula breaks. For k passes out of n, with p = k/n and z = 1.96 for 95%:

centre = (p + z²/2n) / (1 + z²/n)
half   = z · √( p(1−p)/n + z²/4n² ) / (1 + z²/n)
interval = centre − half  to  centre + half

Notice what happens as n grows: the same 90% gives a much narrower range with 400 cases than with 20.

Your task

Choose one way to do it:

  • Python: implement wilson(). Run prints the intervals for counts.json. On submit it also runs on counts you have not seen, including all-pass and all-fail.
  • No code: use the calculator and enter the four intervals below as percentages with one decimal place.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on counts.json

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Calculator

No-code route: Wilson calculator

Enter passed and run; read the interval. Use it for the boxes below.

Enter whole numbers, with passed no more than run.
3

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is one way to pass "Intervals right for 20 and 100 cases (code or calculator)"; the task explains the other.

4

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is one way to pass "Intervals right for 20 and 100 cases (code or calculator)"; the task explains the other.

5

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is one way to pass "Intervals right for 20 and 100 cases (code or calculator)"; the task explains the other.

6

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is one way to pass "Intervals right for 20 and 100 cases (code or calculator)"; the task explains the other.

7

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is a harder check: it earns the second 50 points once the core checks pass. This is one way to pass "Right at the edges: all passed and none passed"; the task explains the other.

8

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is a harder check: it earns the second 50 points once the core checks pass. This is one way to pass "Right at the edges: all passed and none passed"; the task explains the other.

9

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is a harder check: it earns the second 50 points once the core checks pass. This is one way to pass "Right at the edges: all passed and none passed"; the task explains the other.

10

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is a harder check: it earns the second 50 points once the core checks pass. This is one way to pass "Right at the edges: all passed and none passed"; the task explains the other.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Wilson confidence interval for an eval pass rate: why 18 of 20 is not 90% accurate

AI Testing for QA EngineersIntermediateStatistics for testers

How it works

A pass rate from a small set is an estimate, and an estimate needs a range. With twenty cases, the true rate could be far from the observed one. The Wilson score interval is the usual range for pass rates, because it behaves sensibly with small samples and at the extremes of all passing or all failing, where the textbook formula breaks.

The lesson lets you compute it in Python or with a calculator. Either way, the interesting part is how the width shrinks as the number of cases grows, which is why bigger sets give firmer claims.

About the assignment

Compute Wilson intervals for several pass rates, including the edge cases at zero and all passing.

What you will do

  1. Pick Python or the calculator.
  2. Apply the formula on the page for each result.
  3. Check that the zero and all-pass cases give sensible ranges.

Common mistakes

  • Using the plus-or-minus formula

    At 0% or 100% it gives a range of zero width, which is wrong.

  • Mixing percentages and fractions

    Use fractions in the formula and convert at the end.

In a real test suite

Reporting a pass rate with its interval stops arguments about small differences before they start, and it makes clear when a set is too small to support a claim.

When two intervals overlap a lot, the two versions may well be equally good. That is a useful result, not a failure of the method: it tells the team that a decision needs more cases, a paired comparison, or a different argument altogether.

Related lessons

Next lessonFind your noise floorAll lessons in the courseAI Testing for QA Engineers