Skip to content
CodeKruPro

Same cases, paired comparison

44/99

Module · Statistics for testers

intermediate

about 35 min

What this lesson teaches

The idea

When two versions run on the same cases, most cases pass on both or fail on both, and those say nothing about which version is better.

Why a tester cares

Only the cases that changed matter. Counting them gives a fairer comparison than two headline pass rates.

Words you will meet

Paired comparison:
comparing two versions case by case on the same test set.
Discordant case:
a case that passed on one version and failed on the other.
McNemar's test:
the standard test for paired pass or fail results.

What you will do

  1. 1Count discordant cases
  2. 2Compute chi-square and p
  3. 3Give a verdict
  4. 4Submit
Jump to practice

When both versions run on the same cases, comparing their two pass rates as if they were independent throws information away. Most cases pass on both or fail on both; they say nothing about which version is better. Only the discordant cases count: b cases that passed before and fail now, and c cases that failed before and pass now.

McNemar's test uses just those two numbers:

chi² = (|b − c| − 1)² / (b + c)        (with continuity correction)
p    = erfc( √(chi² / 2) )             (one degree of freedom)

Your task

The new version scored 71% and the old 69% on the same 200 cases. Complete the script, and say whether there is a detectable difference. On submit it also runs on another pair of versions.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on pairs.csv

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

McNemar test for paired LLM eval comparison on the same cases

AI Testing for QA EngineersIntermediateStatistics for testers

How it works

When two versions run on the same cases, comparing their pass rates as if they were independent throws information away. Most cases pass on both or fail on both, and those say nothing about which version is better.

McNemar's test uses only the discordant cases: those that passed before and fail now, and those that failed before and pass now. If the two counts are close, the versions are not detectably different, whatever the headline rates suggest.

About the assignment

Count the discordant cases between two versions on the same 200 cases, compute the test statistic and probability, and say whether there is a detectable difference.

What you will do

  1. Pair each case's old and new result.
  2. Count the two kinds of discordant case.
  3. Apply the formula on the page and give a verdict from the probability.

Common mistakes

  • Counting concordant cases

    Cases with the same result in both versions do not enter the test.

  • Forgetting the continuity correction

    The formula subtracts one inside the square. Leave it in for small counts.

In a real test suite

Paired comparisons are the norm for prompt and model changes, because both versions can always run on the same frozen set, which is the cheapest way to get a sensitive comparison.

The discordant cases are also the most useful ones to read by hand: they show exactly what the change fixed and what it broke.

If the discordant counts are small, the probability will be large even when one version looks better on paper. That is the test saying there is not enough evidence yet, and the fix is more cases that separate the versions, not a different statistic.

Related lessons

Next lessonHow many cases do you need?All lessons in the courseAI Testing for QA Engineers