Skip to content
CodeKruPro

CI tiers with a cost budget

81/99

Module · Release, production and operations

advanced

about 45 min

What this lesson teaches

The idea

Not every eval should run on every change. Cheap checks run on every pull request; expensive ones run only when they can tell you something new.

Why a tester cares

Splitting tests into tiers, and caching results, keeps the monthly bill inside budget without losing the checks that matter.

Words you will meet

Pull request:
a proposed code change, reviewed before it is merged.
Tier:
a group of tests that run at the same point, such as on every change or before a release.
GitHub Actions:
GitHub's CI system, configured in YAML files.

What you will do

  1. 1Read the suites
  2. 2Write the tiers
  3. 3Retire one check
  4. 4Cost the plan
Jump to practice

Not every eval should run on every change. Cheap checks run on every pull request, expensive ones only when they can tell you something. A common split has three tiers:

  • PR checks on every pull request: seconds, a few rupees.
  • Prompt change when a pull request touches the prompts: the golden set and safety.
  • Pre-release on a release tag or a manual run: the full judge, load and safety.

Two more habits keep the bill down. Cache eval results keyed on the prompt and dataset files, so a re-run of the same inputs costs nothing. And retire a check that has passed every time for months and is covered elsewhere: it costs money and tells you nothing.

Your task

  1. Finish ci.yml from the suite list. Run shows how a reviewer's script reads it.
  2. Fill in the cost sheet. The plan must fit the budget.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
Suites, tiers and budgetShow
Eval suites (cost is one run, in rupees; history is the last 90 days)

suite        cost   runs   pass rate   what it guards
smoke           2    420     97.4%     answers parse; key facts present
format_json     3    400    100.0%     reply is valid JSON (also covered by smoke's schema check)
golden         40     61     93.1%     the 120-case golden set
safety         25     65     99.2%     hard-gate cases: refusals, limits, PII
judge_full    300      9     91.0%     LLM-judge scoring of 500 conversations
load          150      9    100.0%     50 concurrent requests, p95 and error rate

Tiers
  pr-checks      every pull request          cheap, fast checks
  prompt-change  pull requests that touch prompts/**   the golden set and safety
  pre-release    a release tag (v*) or a manual run      safety, judge_full and load

Volumes per month: 120 pull requests, of which 15 touch prompts/**, and 4 releases.
Budget: Rs 3,200 a month. Retire one check that has stopped telling you anything.
1

Write it up

How this is checked: When you submit, it is saved as ci.yml and used by the run.

2

Your code

Provided by the lesson. You can read it but not change it.

on your ci.yml

How this is checked: When you submit, your script runs and the server compares what it prints with the right values.

3

Write it up

One row per tier: runs a month, and the cost of one run of everything in that tier.

How this is checked: When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

GitHub Actions tiers for LLM evals with a cost budget: what runs on every PR and what waits

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

Not every eval should run on every change. Cheap checks run on every pull request; expensive ones run only when they can tell you something. A common split has three tiers: quick checks on every pull request, the golden set and safety suite when prompts change, and the full judge, load and safety runs before a release.

GitHub Actions expresses this with triggers and job conditions. A paths filter job reports whether prompt files changed, and later jobs depend on it. Release jobs run on a tag push or a manual dispatch.

Two more habits keep the bill down. Cache eval results keyed on the prompt and dataset files, so re-running identical inputs costs nothing. Retire a check that has passed every time for months and is covered elsewhere, because it costs money and tells you nothing new.

About the assignment

Finish the workflow file from the suite list so each tier runs the right suites under the right conditions, then cost the plan against the monthly budget.

What you will do

  1. Add the release triggers under the workflow's trigger section.
  2. Make the prompt-change job depend on the changes job and run only when prompts changed.
  3. Add caching to the expensive tiers and drop the saturated check.
  4. Multiply each tier's runs a month by the cost of its suites, and check the total.

Common mistakes

  • Running the full judge on every pull request

    It may fit the budget once, but it is the first thing to break the budget as pull requests grow.

  • Retiring a check that guards something unique

    Only retire a check that is both saturated and covered by another one.

In a real test suite

The reviewer's script in this lesson reads the YAML the way a teammate would. Teams sometimes keep a similar check in CI itself, so a workflow edit that drops the safety suite fails before it merges.

Related lessons

Next lessonA model upgrade: what blocks itAll lessons in the courseAI Testing for QA Engineers