Skip to content
CodeKruPro

Build a tiny eval harness

15/99

Module · A deterministic harness

beginner

about 45 min

What this lesson teaches

The idea

An eval harness runs a whole set of test cases against a bot and summarises the results, broken down by kind of failure.

Why a tester cares

One pass rate hides whether the losses are wrong facts or broken promises, which need different fixes. Writing a small harness shows what tools like DeepEval do for you.

Words you will meet

Eval:
a set of test cases run against an AI feature to measure how well it behaves.
Harness:
the code that runs the cases and collects the results.
Failure mode:
one kind of failure, such as "wrong refund time".

What you will do

  1. 1Read cases.jsonl
  2. 2Write the checks
  3. 3Print the report
  4. 4Submit
Jump to practice

A single test tells you pass or fail. An eval harness runs a whole dataset and tells you how many cases pass, and which kinds of failure are growing. Every team that ships AI features runs one; frameworks like DeepEval and Promptfoo are harnesses with more features. Writing a small one yourself is the fastest way to understand them.

cases.jsonl holds one case per line: an id, the question, a must_match regular expression, sometimes a must_not_match, and the failure mode the case belongs to.

Your task

Complete eval.py. For each case, ask RefundBot the question and decide pass or fail: must_match must be found and must_not_match, when present, must not be. Ignore case in both. Then print one line of JSON:

{"total": 11, "passed": 10, "pass_rate": 0.909, "by_mode": {"wrong_fact": {"total": 4, "failed": 0}, ...}}

Round pass_rate to three decimals. On submit your harness runs on a faulty build and on a larger dataset you have not seen, and its numbers are compared with the right ones.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

against the practice bot

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Build a simple LLM eval harness in Python that reports pass rates by failure mode

AI Testing for QA EngineersBeginnerA deterministic harness

How it works

An eval harness runs a whole case set and summarises it: how many cases passed, and which kinds of failure are growing. Frameworks such as DeepEval and Promptfoo are harnesses with more features. Writing a small one shows exactly what they do for you.

The key design choice is reporting by failure mode. A single pass rate hides whether the losses are wrong facts, broken promises or missing contacts, and those need different fixes from different people.

About the assignment

Complete eval.py so it asks RefundBot each question in the case file, applies its patterns, and prints one JSON line with totals, a rounded pass rate and a per-mode breakdown.

What you will do

  1. Load every case from the file at run time.
  2. Pass a case only when its must-match pattern is found and its must-not-match pattern, if any, is not.
  3. Build the per-mode table in the same loop and print the report once.

Common mistakes

  • Ignoring must-not-match

    A reply can contain the right words and still promise something false. That second pattern is what catches it.

  • Hard-coding the report

    Grading runs on a larger case file and another build, so printed numbers must come from the run.

In a real test suite

The harness you write here returns in later modules: in DeepEval in lesson 89, in Java or JavaScript in lesson 91, and as the source of numbers for release gates in Module 14.

Related lessons

Next lessonTest the test: false alarms and missesAll lessons in the courseAI Testing for QA Engineers