Skip to content
CodeKruPro

Production sampling

86/99

Module · Release, production and operations

advanced

about 40 min

What this lesson teaches

The idea

In production you cannot grade every conversation with an expensive AI judge, and you do not need to.

Why a tester cares

Cheap checks run on everything, a judge runs on a sample, an alarm watches the daily fail rate, and real failures join the regression set.

Words you will meet

Sampling:
checking a random share instead of everything.
Drift alarm:
an alert when the fail rate rises well above its usual level.
Regression set:
test cases built from real past failures.

What you will do

  1. 1Check every row
  2. 2Find the drift
  3. 3Size the sample
  4. 4Promote failures
Jump to practice

In production you cannot judge every conversation with an expensive model, and you do not need to. Split the work:

  • Cheap checks on everything: empty replies, cut-off replies, and simple rules such as "a refund-window answer states 30 days". They cost almost nothing.
  • A judge on a sample: as many conversations as the budget allows.
  • Promotion: when a real conversation fails, it becomes a regression case, so the next release is tested on what customers actually asked.

Watch the daily fail rate. A drift alarm fires on the first day whose rate is above 0.10 and more than twice the mean of the first three days.

Your task

  1. Write the cheap checks and the drift alarm in sample.py.
  2. Work out what share of traffic the judge budget covers.
  3. Promote at least three failing logged questions to promoted.jsonl. On submit they run against the current bot, which must pass them, and a drifted one, which must fail at least one.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on the log, then your cases

How this is checked: When you submit, your script runs on 3 sets of data, 2 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Your answer

How this is checked: Your number must match the right value; small rounding differences are fine.

3

Write it up

How this is checked: When you submit, it is saved as promoted.jsonl and used by the run.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Production monitoring for LLM chatbots with sampling: cheap checks, judge budget and drift alarms

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

In production you cannot judge every conversation with an expensive model, and you do not need to. Cheap checks run on everything: empty replies, cut-off replies, and simple rules such as a refund-window answer stating the right number. A judge runs on a sample as large as the budget allows.

A drift alarm watches the daily fail rate from the cheap checks and fires on the first day that is both above a floor and well above the early baseline. When real conversations fail, they are promoted to the regression set, so the next release is tested on what customers actually asked.

About the assignment

Write the cheap checks and the drift alarm, size the judge sample from the budget, and promote failing logged questions to regression cases that pass on the current bot and catch the drifted one.

What you will do

  1. Implement each cheap check exactly as the docstring defines it.
  2. Compute daily fail rates and find the first day that trips the alarm.
  3. Copy failing questions from the log, with patterns any correct answer passes.

Common mistakes

  • Inventing cases

    Promoted cases must come from failing log rows. Made-up questions miss what customers really asked.

  • Patterns that only match one wording

    A promoted case that fails a correct bot will be deleted by the next person who sees it.

In a real test suite

Sampling plans are budget documents as much as test documents. Writing down the sample rate and its cost makes it easy to argue for more when an incident shows the sample was too thin.

Related lessons

Next lessonShip or hold with conflicting evidenceAll lessons in the courseAI Testing for QA Engineers