Skip to content
CodeKruPro

Build a gate script

80/99

Module · Release, production and operations

advanced

about 40 min

What this lesson teaches

The idea

A gate is a script that runs after the evals in CI. Its exit code decides whether the build may continue.

Why a tester cares

A good gate blocks real regressions and hard failures and lets noise through, or the team learns to ignore it.

Words you will meet

CI:
continuous integration: the automatic build and test run on every change.
Exit code:
the number a script ends with; 0 means success, anything else fails the build.
Noise band:
how far a result may move before it counts as a real change.

What you will do

  1. 1Check the case ids
  2. 2Check hard-gate cases
  3. 3Compare with the noise band
  4. 4Submit
Jump to practice

A gate is a script CI runs after the evals. Its exit code decides the build: 0 lets it through, 1 blocks it. A good gate blocks real regressions and hard failures, and does not block on noise, or the team learns to ignore it.

Each results file has every case with one True or False per repeated run, and a hard flag on hard-gate cases. The rules:

  1. If the two files do not have the same case ids, exit 2: the results cannot be compared.
  2. If any hard-gate case fails in any current run, exit 1.
  3. Work out the pass rate of each run. The noise band is twice the spread of the baseline's runs (highest minus lowest), and never less than 0.02. If the current mean is lower than the baseline mean by more than the band, exit 1.
  4. Otherwise exit 0.

Your task

Write gate.py. Run shows its exit code on a clean result and a regressed one. On submit it also faces a noisy result, a failed hard-gate case, a tight baseline and a broken results file.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on two result sets

How this is checked: When you submit, your script runs on 6 sets of data, 4 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Release gate script for LLM evals in CI: exit codes for regressions, hard fails and noise

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

A gate is a script CI runs after the evals; its exit code decides the build. A good gate blocks real regressions and hard failures and lets noise through, or the team learns to ignore it.

The rules here are explicit. Results that cannot be compared, because their case sets differ, get their own exit code. Any failing hard-gate case blocks. Otherwise the gate compares the mean pass rate with the baseline's, allowing a noise band based on how much the baseline's own runs varied, with a minimum.

About the assignment

Write a gate that exits with the right code on clean, regressed, noisy, hard-fail, steady-baseline and broken result files.

What you will do

  1. Check that both files have the same case IDs before anything else.
  2. Block on any hard-gate case that failed in any run.
  3. Compute each run's rate, the band from the baseline's spread, and compare the means.

Common mistakes

  • A fixed threshold

    A threshold that ignores the baseline's spread blocks noise on a jumpy bot and misses drops on a steady one.

  • Silent comparison of different case sets

    Different cases make the means meaningless. Refuse with a distinct exit code.

In a real test suite

Print one line explaining the decision, with the numbers behind it. Engineers reading a failed CI job should not have to rerun anything to understand why it failed.

Test the gate itself with fixture files, as this lesson does, before trusting it in CI. A gate that has only ever seen real results has never been shown a regression, so nobody knows whether it can block one.

Related lessons

Next lessonCI tiers with a cost budgetAll lessons in the courseAI Testing for QA Engineers