A test is only as good as its idea of "correct", and testers are often not the right people to decide it. Before you write a rubric or a gold set, compare your labels with someone who knows the domain. Where you disagree, someone has to decide, and it is not always the same person.
Below, a tester and Acme's support lead (the domain expert) labelled the same 15 replies.
Your task
Count how many replies they agree on, and give the agreement as a percentage.
For four kinds of dispute, choose who should decide.
Draft one pass or fail question for a rubric that would have settled one of the disputes. It is not graded; compare it with the model answer.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
ID Tester Support lead Reply
E01 pass pass Reply quotes the 30-day window and the 48-hour damage rule.
E02 pass fail Reply offers a goodwill voucher the policy does not mention.
E03 pass pass Reply says refunds take 5 to 7 business days.
E04 pass pass Reply gives the support email to a customer asking for a human.
E05 fail fail Reply counts the window from the order date.
E06 fail pass Reply is accurate but uses an emoji in a complaint thread.
E07 fail pass Reply says opened items can be returned.
E08 pass pass Reply refuses to discuss another customer's order.
E09 pass pass Reply says gift cards cannot be refunded.
E10 pass fail Reply waives the Rs 99 pickup fee for a long-time customer.
E11 pass pass Reply answers in Hinglish to a Hinglish question.
E12 pass pass Reply shares the order's delivery address back to the customer.
E13 pass pass Reply says damaged items need a photo within 48 hours.
E14 fail fail Reply quotes a 14-day window.
E15 fail fail Reply is three sentences; the style guide says two.
1
Your answer
How this is checked: Your number must match the right value.
2
Your answer
One decimal place is enough.
How this is checked: Your number must match the right value; small rounding differences are fine.
3
Label each one
Who should settle each kind of dispute?
0 of 4 labelled
1E02 and E10: the reply was generous beyond the written policy. Should goodwill offers pass?
2E07: does "most items can be returned" include opened items?
3E12: both passed it, but may the bot read a customer's address back to them under Acme's privacy notice?
4E15: the style guide says two sentences and the reply has three.
How this is checked: Each row has a right label. All 4 must match. This is a harder check: it earns the second 50 points once the core checks pass.
4
Write it up
Not graded. Compare with the model answer after you submit.
How this is checked: Not graded. Compare it with the model answer after you submit.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Inter rater agreement for LLM evaluation labels: who decides what correct means
AI Testing for QA Engineers·Intermediate·Read before you measure
How it works
A test is only as good as its idea of correct, and testers are often not the right people to decide it. Comparing a tester's labels with a domain expert's on the same replies shows how far apart they are, and agreement as a percentage puts a number on it.
Disagreements are information. Some are settled by the policy text, some need the expert, and some show that nobody has written down what correct means yet. A rubric question that would have settled a dispute is the fix that lasts.
About the assignment
Count how often a tester and Acme's support lead agree on fifteen replies, express it as a percentage, choose who should decide four kinds of dispute, and draft one rubric question.
What you will do
1Compare the two label columns row by row.
2Divide agreements by the total for the percentage.
3For each kind of dispute, ask who owns the knowledge that settles it.
Common mistakes
Assuming the expert is always right
Policy wording or a legal question can override an expert's habit.
Writing a vague rubric question
A good question has a yes or no answer that two people would apply the same way.
In a real test suite
Before building a gold set or calibrating a judge, teams run exactly this exercise. Low agreement between humans means any automatic grader will look worse than it is, or better.
Agreement on its own can mislead when most replies are easy. Teams often look at the disputed replies first: those are where the rubric is unclear, and where an automatic judge is most likely to disagree with people later.
