Skip to content
CodeKruPro

Train, dev and test splits for judges

51/99

Module · Judges and guardrails as classifiers

intermediate

about 35 min

What this lesson teaches

The idea

If you tune a judge's rubric on the examples you read, it will agree with people on those examples and less well on new ones.

Why a tester cares

Split labelled examples three ways: some to write the rubric from, some to choose between versions, and some kept back to report the result once.

Words you will meet

Overfitting:
tuning so closely to known examples that new ones do worse.
Train, dev and test split:
three separate sets: for writing, for choosing, and for reporting.

What you will do

  1. 1Spot the leak
  2. 2Choose on dev
  3. 3Report on test
  4. 4Submit
Jump to practice

Rubrics get tuned. Someone reads failing examples, adds a clause, and the judge agrees with humans on those examples. The danger is the same as in machine learning: a rubric tuned on the examples you read is overfit to them, and its agreement there says little about new replies.

The fix is the usual one. Split labelled examples three ways: train (read them while writing the rubric), dev (choose between versions), and test (report once, and never tune on it).

Your task

Four rubric versions graded 150 labelled replies. The starter chooses a version on the wrong split. Fix it: choose on dev, then report the chosen version's TPR and TNR on test. On submit it also runs on another team's data.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on rubric_versions.csv

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Train, dev and test split for LLM judge rubrics: avoid tuning on the examples you report

AI Testing for QA EngineersIntermediateJudges and guardrails as classifiers

How it works

Rubrics get tuned. Someone reads failing examples, adds a clause, and the judge agrees with humans on those examples. A rubric tuned on the examples you read is overfit to them, and its agreement there says little about new replies.

The fix is the one machine learning uses. Split labelled examples three ways: train for reading while writing the rubric, dev for choosing between versions, and test for reporting once and never tuning on.

About the assignment

Fix a script that chooses a rubric version on the wrong split: choose on dev, then report the chosen version's rates on test.

What you will do

  1. Find where the starter uses the test split to choose.
  2. Choose the best version by its dev results.
  3. Report only that version's rates on test.

Common mistakes

  • Choosing on test

    Picking the version that looks best on test makes the test result optimistic.

  • Reporting several test numbers

    Report the chosen version once. Reporting every version on test turns it into a dev set.

In a real test suite

Teams store the split assignment with the labelled data, so everyone tunes and reports on the same examples and results stay comparable over time.

Keep the test split small but representative, and lock it. Many teams store it separately with a note that nobody reads its replies while working on the rubric, because even looking at failures there quietly turns it into training data.

If the dev and test results of the chosen version differ a lot, that gap is itself a finding: the rubric is sensitive to which replies it sees, and it probably needs simpler wording before anyone relies on it.

Related lessons

Next lessonCorrect an estimate, catch judge driftAll lessons in the courseAI Testing for QA Engineers