Module · Career, capstones and the Cold Product Test
advanced
about 25 min
What this lesson teaches
The idea
Model choices are often argued with benchmark scores. A score only supports a decision about your product if three things hold.
Why a tester cares
Ask whether the questions could be in the model's training data, whether the benchmark measures what your product does, and whether the gap is bigger than chance.
Words you will meet
Benchmark:
a public test set used to compare models.
Contamination:
test questions that appeared in the model's training data.
Model choices are often argued with benchmark scores. Before a score supports a decision about your product, ask three questions:
Contamination: could the test questions be in the model's training data? Public benchmarks published long before a model's cutoff, or questions the model completes word for word, are warning signs.
Mismatch: does the benchmark measure what your product does? A coding score says little about Hinglish customer support.
Noise: is the gap bigger than chance? A fraction of a point on a few hundred questions, or the best of many submissions, is not a difference.
A claim is credible for a decision when it uses data like yours that the model cannot have seen, with a gap bigger than its uncertainty.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
1Model X scores 92% on a public benchmark whose questions were published two years before its training cutoff.
2Model Y tops a coding leaderboard, so it will be the best choice for our Hinglish refund bot.
3Model Z beats Model W by 0.4 points on a 500-question benchmark, from one run each.
4On 300 of our own held-out refund conversations, graded by our calibrated judge, the new model gains 9 points and the 95% intervals do not overlap.
5A vendor reports its best score after submitting 40 times to a public leaderboard.
6The model completes benchmark questions word for word when given their first half.
7Top score on a maths olympiad benchmark, quoted in a pitch for our customer support assistant.
8An independent private benchmark with fresh questions every quarter shows the same 6-point lead over three releases.
How this is checked: Each row has a right label. At least 6 of 8 must match. For the full score, a harder check: all 8 must match.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
LLM benchmark contamination, product mismatch and leaderboard noise: judging a benchmark claim
AI Testing for QA Engineers·Advanced·Career, capstones and the Cold Product Test
How it works
Model choices are often argued with benchmark scores. Before a score supports a decision about your product, ask three questions. Could the test questions be in the model's training data? Does the benchmark measure what your product does? Is the gap bigger than chance?
A claim is credible for a decision when it uses data like yours that the model cannot have seen, with a gap larger than its uncertainty. Your own held-out conversations, graded by a calibrated judge, usually beat any public number.
About the assignment
Classify eight benchmark claims as contamination risk, product mismatch, noise or credible.
What you will do
1Ask where the questions came from and when they were published.
2Compare what the benchmark measures with what your product does.
3Compare the gap with the number of questions and the number of tries.
Common mistakes
Treating the best of many submissions as signal
Choosing the luckiest run is a form of noise.
Calling every public benchmark contaminated
Fresh private benchmarks with stable leads across releases can be credible.
In a real test suite
When someone proposes a model change on benchmark grounds, the strongest response is to run your own frozen set on both models and compare segments, as lesson 82 does.
Writing down the three questions in the team's model-selection template keeps benchmark arguments short and focused on evidence.
Benchmark claims in vendor material often omit the details these questions need. Asking for the question source, the number of runs and the date of the test set is reasonable, and a vendor who cannot answer has told you something useful.
