If most replies pass, two raters who guess at random still agree most of the time. Raw agreement flatters them.
Why a tester cares
Cohen's kappa removes agreement that happens by chance. A judge should be compared with what two people reach using the same rubric, not with perfection.
Words you will meet
Cohen's kappa:
agreement corrected for chance: 0 is no better than guessing, 1 is perfect.
Human ceiling:
how well two people agree; a judge cannot be expected to beat it.
Raw agreement flatters raters: if 70% of replies pass, two raters who guess at random will agree more than half the time. Cohen's kappa corrects for that:
0 means no better than chance, 1 means perfect. The useful comparison is not "is the judge's kappa high?" but "is it close to what two humans reach with the same rubric?". Humans set the ceiling. A judge cannot be held to agreement the rubric does not allow.
Your task
Implement kappa, compute it for two humans and for the judge against one human, and decide what to improve first.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
15"humans": round(kappa([r["rater_1"]for r in humans],[r["rater_2"]for r in humans]),3),
16"judge_vs_human": round(kappa([r["human"]for r in judge],[r["judge"]for r in judge]),3),
17}))
18
on both files
How this is checked: When you submit, your script runs and the server compares what it prints with the right values.
2
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Cohen kappa for LLM judge agreement, and the human ceiling it should be compared with
AI Testing for QA Engineers·Intermediate·Judges and guardrails as classifiers
How it works
Raw agreement flatters raters. If most replies pass, two raters guessing at random still agree often. Cohen's kappa corrects for agreement expected by chance: zero means no better than chance, one means perfect.
The useful comparison is not whether a judge's kappa is high, but whether it is close to what two humans reach with the same rubric. Humans set the ceiling, and a judge cannot be held to agreement the rubric itself does not allow.
About the assignment
Implement kappa, compute it for two humans and for the judge against one of them, and choose what to fix.
What you will do
1Compute observed agreement and each rater's pass share.
2Compute chance agreement and then kappa, as the page defines.
3Compare the judge with the human pair before deciding what to fix.
Common mistakes
Blaming the judge for a weak rubric
If humans disagree too, fix the rubric first.
Dividing by zero
When chance agreement is one, kappa is undefined. Handle it.
In a real test suite
Teams report a judge's kappa next to the human pair's. When the two are close, the judge is as good as the rubric allows, and more prompt work on the judge will not help.
Kappa is sensitive to how common passes are, so compare values only on the same set of replies. A judge measured on an easy week and humans measured on a hard one will give numbers that cannot be compared fairly.
