Skip to content
CodeKruPro

One run is not evidence

5/99

Module · Why exact match fails on AI

beginner

about 25 min

What this lesson teaches

The idea

Some faults show up on only some replies. Asking once can land on a good reply and pass, so the test misses a broken bot.

Why a tester cares

A little probability tells you how many times to ask before a miss becomes rare enough to accept.

Words you will meet

Miss:
a test that passes on a faulty bot.
Fault rate:
the share of replies that come back wrong, such as 4 in 10.
Repeats:
asking the same question several times in one test.

What you will do

  1. 1Work out the odds
  2. 2Choose the repeats
  3. 3Submit
Jump to practice

Some faults only appear on some replies. Suppose a build of RefundBot gives the wrong return window on 40% of replies and the right one on the other 60%. A test that asks once will pass whenever it happens to get a good reply.

Your task

  1. Work out the chance that a test asking once misses this fault, then the chance when it asks 3 times and 10 times. Each reply is independent, and the test fails if any reply is wrong.
  2. Find the smallest number of asks that brings the chance of missing it below 1 in 1,000.
  3. Set REPEATS in the test to that number or more, and submit. Your test runs against a correct bot and against the faulty build several times, with different random seeds.

Give probabilities as decimals, for example 0.25 rather than 25%. Three decimal places are enough.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine.

2

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine.

3

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine.

4

Your answer

How this is checked: Your number must match the right value.

5

Your code

against the practice bot

How this is checked: When you submit, your tests run against 7 versions of RefundBot: 2 correct and 5 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

How many times to repeat a flaky LLM test before a missed fault becomes unlikely

AI Testing for QA EngineersBeginnerWhy exact match fails on AI

How it works

Some faults show up on only a share of replies. If a build gives a wrong answer on some fraction of calls, a test that asks once will often see a good reply and pass. That is a miss, and it looks exactly like a healthy build.

Probability turns this into a design decision. If each reply is independent, the chance that every one of n replies is good is the good-reply rate multiplied by itself n times. Asking more often drives that chance down quickly, and you can choose the smallest number of repeats that makes a miss rare enough for your risk.

About the assignment

You work out the chance that a single ask, three asks and ten asks all miss a fault that appears on part of the replies, find the smallest number of repeats that makes a miss rarer than one in a thousand, and set the test to use it.

What you will do

  1. Write the miss chance for one ask as a decimal, then raise it to the power of the number of asks.
  2. Keep increasing the repeats until the miss chance drops below the threshold.
  3. Set the repeat constant in the test, and check it still finishes quickly on a correct bot.

Common mistakes

  • Mixing up the two rates

    The chance of a miss uses the share of good replies, not the share of bad ones.

  • Percentages in the answer boxes

    The boxes expect decimals between 0 and 1.

In a real test suite

Repeat counts cost time and, with real models, money. Teams set them per check by risk: a few repeats for wording checks, many more for checks that guard money, safety or personal data, where a miss is expensive.

Related lessons

Next lessonMake the bot leak a secretAll lessons in the courseAI Testing for QA Engineers