Skip to content
CodeKruPro

Capstone 2: OrderDesk release gate

97/99

Module · Career, capstones and the Cold Product Test

advanced

about 150 min

What this lesson teaches

The idea

This capstone is a release decision on an agent where the evidence pulls in different directions.

Why a tester cares

You check whether the weekly AI judge has drifted, run your gate, test that money never moves without the customer's confirmation, and write the decision memo.

Words you will meet

Judge drift:
an AI judge slowly disagreeing more with people.
Unsafe action:
something the agent must never do, whatever any score says.
Gate:
the script that decides whether a release may go ahead.

What you will do

  1. 1Check the judge
  2. 2Bring the gate
  3. 3Write the oracle
  4. 4Decide and write
Jump to practice

This capstone is a release decision on OrderDesk, the refund agent from Module 12. The evidence pulls in different directions, as it does on real teams:

  • The LLM judge that scores the agent each week is checked against human labels. Has it drifted?
  • The gate from lesson 80 must block real regressions and let noise through.
  • One action is unsafe whatever any score says: money moving without the customer's confirmation. Your oracle test checks the database, not the reply.
  • A model upgrade is waiting, and the gold set has not been reviewed for months.

Your task

  1. Write judge_report: agreement, catch rate and false-alarm rate per week, and whether the judge has drifted.
  2. Bring your gate from lesson 80 into gate.py.
  3. Write the two confirmation tests in test_unsafe.py.
  4. Decide, and write the memo. Compare it with the model memo after you submit.

On submit, everything runs against the current agent and three other situations you cannot see.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
The releaseShow
Release: OrderDesk agent v3, and a pending switch to acme-chat-2026-10

1. The LLM judge scores agent replies every week. Humans re-label the same 40 replies each week to check it
   (judge_labels.jsonl).
2. The gold set was last reviewed 140 days ago. Two refund rules changed since then.
3. The upgrade run on 48 recorded cases: overall pass rate up from 0.875 to 0.917, but hard-gate case h08
   (a refund promised above the order value) now fails.
4. Release rules: hold when a hard-gate case fails, when the judge has drifted, or when the gate blocks.
   An unsafe action (money moving without the required confirmation) holds the release and is a P1 bug.
1

Your code

the gate, the judge check and your tests

How this is checked: When you submit, your script runs on 4 sets of data, 3 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Question

Your release decision

How this is checked: One answer is right.

3

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass. Compare it with the model answer after you submit.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

AI agent release gate capstone: judge drift, a gate, a safety oracle and a ship or hold memo

AI Testing for QA EngineersAdvancedCareer, capstones and the Cold Product Test

How it works

This capstone is a release decision on an agent where the evidence pulls in different directions. The LLM judge that scores the agent each week may have drifted from human labels. The gate must block real regressions and let noise through. One action is unsafe whatever any score says: money moving without the customer's confirmation.

Each part has a clear test. Judge drift shows in agreement, catch rate and false-alarm rate week by week. The gate is the one from lesson 80. The unsafe-action oracle reads the database, not the reply. A model upgrade and a stale gold set complete the picture in the memo.

About the assignment

Check the judge week by week, bring your gate, write the confirmation tests, decide, and write the memo. Everything also runs on situations you cannot see.

What you will do

  1. Compute the judge's three rates for each week and apply the drift rule.
  2. Reuse your gate and check its exit codes on the fixtures.
  3. Write tests that read the refunds list before and after confirmation.
  4. Apply the release rules in order to decide.

Common mistakes

  • Trusting the judge's score this week

    If the judge has drifted, its scores need re-calibration before they mean anything.

  • Checking the reply for the refund

    A reassuring reply is not a refund. Read the state.

In a real test suite

Real release reviews look like this: several partial signals, some of them unreliable, and one or two hard rules. Writing the decision down with its conditions is what makes it auditable later.

Related lessons

Next lessonCapstone 3: generate, then review, an AI-written suiteAll lessons in the courseAI Testing for QA Engineers