Skip to content
CodeKruPro

Write a binary rubric

47/99

Module · Judges and guardrails as classifiers

intermediate

about 35 min

What this lesson teaches

The idea

An LLM judge is an AI model asked to grade another model's replies. A 1 to 5 score sounds precise but nobody can act on it.

Why a tester cares

A judge that asks one yes or no question about one kind of failure gives results you can count, compare with people and trust.

Words you will meet

LLM judge:
an AI model used to grade replies.
Rubric:
the written instructions the judge grades by.
Binary:
with only two answers, pass or fail.

What you will do

  1. 1Read the current judge's scores
  2. 2Predict five more
  3. 3Rewrite the rubric
  4. 4Submit
Jump to practice

An LLM judge is a model asked to grade another model's output. Most teams start with a rubric like "rate from 1 to 5 for helpfulness and accuracy". It feels thorough and produces numbers nobody can act on. Is a 3 a failure? What changed when the average went from 3.8 to 3.6? And a 1-to-5 judge mixes things up: asked about "helpful and accurate", it tends to reward long, warm replies.

A good rubric asks one pass-or-fail question about one failure mode, defines both answers, and shows an example of each. Binary verdicts can be counted, compared with human labels, and turned into rates you can test, which the rest of this module does.

Your task

  1. Look at the current judge's scores on ten replies, next to a human's verdict on whether each reply's policy facts are right.
  2. Predict whether the current judge would score five more replies 4 or above (its "pass").
  3. Rewrite the rubric as one binary question about wrong policy facts, on the template.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
The current rubricShow
Rate the support reply from 1 to 5 for how helpful and accurate it is. 5 is excellent. Scores of 4 and 5 count as a pass.
Ten replies: the judge's score and a human's verdict on the factsShow
ID   Score  Human  Reply
J01  2      pass   Short and correct: "Returns are accepted for 30 days after delivery."
J02  5      fail   Long, warm and wrong: three friendly paragraphs saying the window is 60 days.
J03  5      pass   Long, warm and correct: three friendly paragraphs on the 30-day window.
J04  2      fail   Short and wrong: "You have 14 days."
J05  4      pass   Medium, polite, correct, with an apology for the delay.
J06  4      fail   Medium, polite and wrong about the pickup fee.
J07  1      pass   Short, correct, a little blunt: "30 days. Keep the box."
J08  5      fail   Long and confident, invents a "premium 90-day window".
J09  3      pass   Short, correct, cites the policy line.
J10  5      pass   Long, careful, correct, with steps to start the return.
1

Label each one

Would the current judge pass (score 4 or 5) each reply?

0 of 5 labelled

  1. 1Two warm paragraphs that say gift cards can be refunded.
  2. 2One correct sentence: "Gift cards cannot be refunded."
  3. 3A long, polite reply with the right 48-hour damage rule and a free pickup.
  4. 4A blunt, correct reply: "Rs 99 for pickup. Damaged items free."
  5. 5A long, apologetic reply that gets the refund time wrong (2 days).

How this is checked: Each row has a right label. All 5 must match. This is a harder check: it earns the second 50 points once the core checks pass.

2

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Binary rubric for an LLM judge: one pass or fail question instead of a 1 to 5 score

AI Testing for QA EngineersIntermediateJudges and guardrails as classifiers

How it works

An LLM judge is a model asked to grade another model's output. Teams often start with a 1 to 5 rubric for helpfulness and accuracy. It produces numbers nobody can act on: is 3 a failure, and what does a drop from one average to another mean? Such judges also tend to reward long, warm replies over correct ones.

A good rubric asks one pass or fail question about one failure mode, defines both answers, and shows an example of each. Binary verdicts can be counted, compared with human labels, and turned into rates, which the rest of Module 10 does.

About the assignment

Read a 1 to 5 judge's scores, predict five more, and rewrite the rubric as a single binary question with definitions and examples.

What you will do

  1. Look for what the current scores reward that has nothing to do with correctness.
  2. Pick one failure mode and write a yes or no question about it.
  3. Define pass and fail, and give an example reply of each.

Common mistakes

  • Two questions in one

    A rubric that asks about accuracy and tone at once cannot be scored consistently.

  • No examples

    Without examples, the judge and humans read the definitions differently.

In a real test suite

Teams keep one judge per failure mode rather than one judge for everything. Each small judge can be measured and fixed on its own.

Test the new rubric on a handful of labelled replies before using it at scale. If you and a colleague cannot apply it the same way to those examples, the judge will not either, and the definitions need another pass.

Related lessons

Next lessonA judge is a classifier: TPR and TNRAll lessons in the courseAI Testing for QA Engineers