A judge is wrong some of the time, so before trusting it you compare its verdicts with human labels on the same replies.
Why a tester cares
Two numbers describe it: how many real failures it catches, and how many good replies it correctly passes. A judge that misses failures is not doing its job.
An LLM judge is a classifier, and like any classifier it is wrong some of the time. Before trusting its numbers, compare it with human labels on the same replies. Two rates describe it:
TPR (true positive rate, also called recall or sensitivity): of the replies humans called failures, the share the judge also flagged.
TNR (true negative rate, or specificity): of the replies humans passed, the share the judge also passed.
Here "fail" is the positive class, because finding failures is the judge's job. A judge with a TPR of 0.6 misses four failures in ten, however good its overall agreement looks.
Your task
Complete the script to count the four outcomes and compute TPR and TNR. On submit it also runs on a held-back set of labels.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
LLM judge true positive rate and true negative rate against human labels
AI Testing for QA Engineers·Intermediate·Judges and guardrails as classifiers
How it works
An LLM judge is a classifier, wrong some of the time. Before trusting its numbers, compare it with human labels on the same replies. Two rates describe it. The true positive rate is the share of human-labelled failures the judge also flagged. The true negative rate is the share of human-labelled passes the judge also passed.
Here failure is the positive class, because finding failures is the judge's job. A judge with a low true positive rate misses many failures, however good its overall agreement looks.
About the assignment
Count the four outcomes of a judge against human labels, and compute its true positive and true negative rates.
What you will do
1Decide the positive class before counting.
2Count true positives, false negatives, true negatives and false positives.
3Compute both rates from those counts.
Common mistakes
Swapping the classes
If pass is treated as positive, both rates change meaning.
Reporting agreement only
High agreement can hide a judge that misses most failures.
In a real test suite
These two rates are reused later: lesson 52 corrects observed failure rates with them, and capstone 2 tracks them week by week to catch judge drift.
Collect the human labels independently of the judge's output, so labellers are not nudged by its verdicts. A small set of carefully labelled replies, refreshed now and then, is enough to keep these rates honest over time.
Expect the two rates to move in opposite directions when a judge prompt is made stricter or looser. Report both every time, because improving one while quietly losing the other is the most common way a judge gets worse after a well-meant edit.
