A judge's flag rate is not the failure rate. Some flags are false alarms and some failures slip through.
Why a tester cares
When a judge is upgraded its flag rate can jump though the product did not change. Correcting for the judge's error rates shows whether anything really moved.
Words you will meet
Flag rate:
the share of replies the judge marked as failures.
Phantom regression:
a drop that comes from the measuring tool changing, not the product.
A judge flags 15% of production replies. That is not the failure rate. Some flags are false alarms and some failures slip through, and you measured how often in lesson 48. With the judge's TPR and TNR you can correct the observed rate:
This matters most when the judge changes. Upgrade the judge to catch more failures and the observed rate jumps, though the product did not change. That is a phantom regression. Compare corrected rates, and only call a change real when the corrected rate moved.
Your task
Implement the correction and run it on four before-and-after weeks.
Label each week real or phantom. Call it real when the corrected failure rate rose by more than 2 points.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
How this is checked: When you submit, your script runs and the server compares what it prints with the right values.
2
Label each one
Real regression or phantom?
0 of 4 labelled
1S1: observed 15% then 21%. The judge was upgraded between the two weeks.
2S2: observed 12% then 20%. Same judge both weeks.
3S3: observed 10% then 13%. The judge was upgraded to catch more failures.
4S4: observed 9% then 18%. A small judge prompt tweak, and a new product release.
How this is checked: Each row has a right label. Every real one must be caught, with no false alarms. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Correct an observed failure rate for judge error, and catch phantom regressions from judge drift
AI Testing for QA Engineers·Intermediate·Judges and guardrails as classifiers
How it works
A judge's flag rate is not the failure rate. Some flags are false alarms and some failures slip through, at rates you can measure against human labels. With the judge's true positive and true negative rates, the observed rate can be corrected to an estimate of the true rate.
This matters most when the judge changes. Upgrade a judge to catch more failures and its flag rate jumps though the product did not change: a phantom regression. Comparing corrected rates shows whether anything really moved.
About the assignment
Implement the correction, apply it to four before-and-after weeks, and label each change real or phantom.
What you will do
1Implement the formula on the page.
2Correct each week with the judge rates that applied that week.
3Call a change real only when the corrected rate moved.
Common mistakes
Using one week's judge rates for both weeks
If the judge changed, each week needs its own rates.
Ignoring impossible results
A correction can fall outside zero to one when rates are near chance. Clamp it and treat the judge as unreliable.
In a real test suite
Teams version their judges like models, and recompute baselines when a judge changes, so a judge upgrade is never mistaken for a product regression.
Always report the judge version next to any weekly failure rate. When a chart shows a jump, the first question should be whether the product, the traffic or the judge changed, and the version number answers the last one immediately.
