Skip to content
CodeKruPro

A prompt change is a deploy

79/99

Module · Release, production and operations

advanced

about 35 min

What this lesson teaches

The idea

A one-line prompt edit can change every answer a product gives, so it deserves what any deploy gets: a record of exactly what ran, and a comparison with a baseline.

Why a tester cares

A comparison only means something when exactly one thing changed. If the model and the prompt both changed, a drop could come from either.

Words you will meet

Run manifest:
a record of the prompt, model, dataset and settings used for a run.
Baseline:
the earlier result you compare against.
Hash:
a short fingerprint of a file that changes when the file does.

What you will do

  1. 1Write the manifest
  2. 2Find what changed
  3. 3Compare pass rates
  4. 4Submit
Jump to practice

A one-line prompt edit can change every answer a product gives, so it deserves what any deploy gets: a record of exactly what ran, and a comparison against a known baseline.

A run depends on four things. Record each in a run manifest:

  • Prompt: its version and a hash of its text.
  • Model: the exact model name, with its date.
  • Dataset: its name and a hash of its contents.
  • Config: temperature, max_tokens, top_p.

A comparison tells you something only when one of the four changed. If the model and the temperature both changed, a drop could come from either, and the result cannot be used.

Your task

  1. Write the manifest for the next run, described below.
  2. Finish compare.py. It prints which variables changed, whether the comparison is valid, both pass rates, the change, and the cases that passed before and fail now. On submit it also runs on two run pairs you have not seen.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Write it up

Next you will test prompt v5 (sha 5be1c0d) on acme-chat-2026-05, against refund-golden (sha 9c2e7aa), with temperature 0, max_tokens 300 and top_p 1.

How this is checked: When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks.

2

Your code

on the recorded runs

How this is checked: When you submit, your script runs on 3 sets of data, 2 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Run manifest for comparing prompt versions: change one variable at a time

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

A one-line prompt edit can change every answer a product gives, so it deserves what any deploy gets: a record of exactly what ran, and a comparison with a known baseline.

A run depends on four things: the prompt, the model, the dataset and the configuration. A run manifest records each, with versions and hashes. A comparison is only meaningful when exactly one of them changed; if the model and the temperature both changed, a drop could come from either.

About the assignment

Write the manifest for the next run, and finish a comparison script that reports which variables changed, whether the comparison is valid, both pass rates, the change and the regressed cases.

What you will do

  1. Fill every manifest field from the description of the next run.
  2. Group manifest fields into the four variables and detect which changed.
  3. Report the comparison as valid only when exactly one variable changed.

Common mistakes

  • Counting field changes

    Prompt version and prompt hash change together. They are one variable.

  • Comparing anyway

    Rates from a two-variable change can still be printed, but must not be marked valid.

In a real test suite

Teams store manifests next to every eval result. When a number surprises someone months later, the manifest is what tells them whether they are comparing like with like.

Generate the hashes from the actual files in CI, rather than typing them, so a manifest cannot claim a prompt version that did not run. The same habit catches a dataset edited in place without a new version.

Related lessons

Next lessonBuild a gate scriptAll lessons in the courseAI Testing for QA Engineers