Skip to content
CodeKruPro

How many cases do you need?

45/99

Module · Statistics for testers

intermediate

about 30 min

What this lesson teaches

The idea

The right number of test cases comes from the smallest change you need to detect, not from a round number.

Why a tester cares

Telling two close pass rates apart needs far more cases than people expect. And zero failures in a small set does not mean failures never happen.

Words you will meet

Sample size:
how many test cases a run needs.
Rule of three:
a quick limit: with zero failures in n cases, the true failure rate is probably below 3 divided by n.

What you will do

  1. 1Size a run
  2. 2Judge 50 cases
  3. 3Apply the rule of three
  4. 4Submit
Jump to practice

Teams often pick 50 or 100 eval cases because those are round numbers. The right number comes from the smallest change you need to detect. To tell a 90% pass rate from 85% with the usual 5% significance and 80% power, each run needs about 686 cases. With 50, a 5-point drop is invisible.

n per run = ( z₁ · √(2p̄(1−p̄)) + z₂ · √(p₁(1−p₁) + p₂(1−p₂)) )² / (p₁ − p₂)²
z₁ = 1.96 (5% two-sided), z₂ = 0.8416 (80% power), p̄ = (p₁ + p₂) / 2

The opposite problem: a run with no failures. Zero of 50 does not mean the failure rate is zero. The rule of three says that with zero failures in n cases, the true rate is below about 3/n with 95% confidence: below 6% for 50 cases.

Your task

Answer the questions with the calculator, or by computing them yourself. Round case counts up.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Calculator

Sample size calculator

Enter two different rates between 0 and 100.
2

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine.

3

Question

Your team runs 50 cases per release. Can it detect that drop?

How this is checked: One answer is right.

4

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine.

5

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is a harder check: it earns the second 50 points once the core checks pass.

6

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine. This is a harder check: it earns the second 50 points once the core checks pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

How many eval cases do you need: sample size for detecting a change, and the rule of three

AI Testing for QA EngineersIntermediateStatistics for testers

How it works

Teams often choose round numbers of eval cases. The right number comes from the smallest change you need to detect. Telling two close pass rates apart with the usual significance and power needs far more cases than intuition suggests, which is why small sets hide real drops.

The opposite problem is a run with no failures. Zero failures in a set does not mean a zero failure rate. The rule of three gives a quick upper bound at 95% confidence, and it shows how many clean cases you need before you can claim a failure is rare.

About the assignment

Size a run for a given change, judge whether a 50-case set can detect it, and apply the rule of three to a run with no failures.

What you will do

  1. Use the sample-size formula or calculator for the two pass rates.
  2. Compare the result with 50 cases.
  3. Apply the rule of three to the zero-failure run.

Common mistakes

  • Sizing for the wrong change

    Use the smallest change that would matter to the product.

  • Reading zero failures as zero risk

    The rule of three bounds the rate; it does not make it zero.

In a real test suite

When the needed size is unaffordable, teams either accept detecting only larger changes, or switch to paired comparisons on the same cases, which need fewer.

Writing the detectable change into the test plan sets honest expectations: a small set catches big regressions, not subtle ones.

Related lessons

Next lessonpass@k, pass^k and severity overridesAll lessons in the courseAI Testing for QA Engineers