Skip to content
CodeKruPro

Find your noise floor

42/99

Module · Statistics for testers

intermediate

about 35 min

What this lesson teaches

The idea

Run the same eval on the same bot twice and the pass rates differ, because models choose their words with some randomness.

Why a tester cares

The spread you see with nothing changed is the noise floor. Any "improvement" smaller than that is not evidence of anything.

Words you will meet

Noise floor:
how much a result moves between runs when nothing has changed.
Run:
one pass through the whole test set.

What you will do

  1. 1Complete one_run()
  2. 2Run five rounds
  3. 3Read the spread
  4. 4Judge the claim
Jump to practice

Run the same eval on the same bot twice and you will usually get two different pass rates. Models sample their words, so some replies pass on one run and fail on the next. The spread you see with nothing changed is your noise floor. Any "improvement" smaller than it is not evidence of anything.

The RefundBot build in this lesson is unchanged between runs, but like a real model it sometimes gets the return window wrong.

Your task

  1. Complete one_run(), run the 10-case eval five times, and print the five pass rates and their spread (highest minus lowest).
  2. Use your noise floor to judge a teammate's claim.

The server keeps its own log of every reply it sent you, and checks your pass rates against it.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

five runs against RefundBot

How this is checked: When you submit, your script runs and the server compares what it prints with the right values.

2

Question

A teammate changed the prompt, ran the eval once, and got 82% instead of last week's 80%. They say the change helped. What do you say?

How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Noise floor of an LLM eval: how much pass rates move with nothing changed

AI Testing for QA EngineersIntermediateStatistics for testers

How it works

Run the same eval on the same bot twice and the pass rates usually differ, because models sample their words. The spread you see with nothing changed is the noise floor. Any claimed improvement smaller than it is not evidence.

The practice server keeps its own log of every reply it sent, so the rates your script prints can be checked against what actually happened, not just against what your script believes.

About the assignment

Complete a function that runs a small eval once, run it several times, print each run's pass rate and the spread, and use it to judge a teammate's claim.

What you will do

  1. Complete one run of the eval so it returns a pass rate.
  2. Repeat it the number of times the lesson asks, and collect the rates.
  3. Compute the spread as highest minus lowest, and compare the claim with it.

Common mistakes

  • Running once

    One run cannot show a spread.

  • Reusing replies between runs

    Each run must send its own requests, or the spread is fake.

In a real test suite

A noise floor turns into a gate rule later: lesson 80 blocks a release only when the drop is bigger than twice the baseline's spread.

The noise floor depends on the bot, the case set and the settings, so measure it again whenever any of those changes. A floor measured on last quarter's model can make a real regression on the new one look like ordinary wobble.

Related lessons

Next lessonNoise or regressionAll lessons in the courseAI Testing for QA Engineers