Skip to content
CodeKruPro

Your harness against a real model

17/99

Module · A deterministic harness

beginner

about 45 min

What this lesson teaches

The idea

Checks written against a scripted bot meet a messier world on a real model. Some checks turn out to fail good replies or pass bad ones.

Why a tester cares

Running your checks on real replies, several times each, shows which ones you can trust before you rely on them.

Words you will meet

Real model:
an AI model from an AI company, such as Claude, here running with RefundBot's instructions.
Misfire:
a check that gets a reply wrong, either way.
Log:
the saved list of every message you sent and every reply you got.

What you will do

  1. 1Send the 20 cases three times
  2. 2Complete the four checks
  3. 3Read what failed
  4. 4Write two notes
Jump to practice

Everything so far ran against a practice bot that we scripted. Real models can be messier: they may hedge, run long, add a phone number you did not expect, or answer a question you asked them not to. Or they may behave well, and some of your checks misfire anyway. This lesson runs your checks on a real model's replies and asks the honest question: which of my checks still hold?

The console sends each of the lesson's 20 cases to a real model with RefundBot's system prompt. Send all 20 cases three times, 60 calls in all. Your log is then written to replies.jsonl whenever you Run or Submit.

Your task

  1. Send all the cases three times (each click of Send all 20 cases sends one round).
  2. Complete the four checks in harness.py and Run it. It prints how many replies pass each check.
  3. Read the replies the checks failed. Name two checks that misfired, either failing a good reply or passing a bad one, and say why. If only one misfired on your log, say so, and describe the reply that would make a second one misfire.

On submit the server applies the same four checks, as this page defines them, to your logged replies and compares counts. One disagreement per check is allowed. Your notes are not graded: compare them with the model answer.

Before a real model is enabled for this course, the console uses a stand-in and says so. The lesson works the same way either way.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Real model with RefundBot's prompt

Real model66 of 66 messages left in this lesson

No messages yet.

Rounds sent: 0 of 3
2

Your code

on your logged replies

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

3

Write it up

Not graded beyond length. Compare with the model answer after you submit.

How this is checked: Not marked for content: it needs at least 40 words. Compare it with the model answer after you submit.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Run LLM eval checks against a real model and find the checks that misfire

AI Testing for QA EngineersBeginnerA deterministic harness

How it works

Checks written against a scripted bot meet a harder world on a real model. Real replies can hedge, run long, add details nobody asked for, or answer questions they were told to decline, and a well-behaved model still exposes checks that were aimed at the wrong cases. Some checks that looked solid turn out to fail good replies or pass bad ones.

Repetition matters even more here. Sending each case several times shows which failures are consistent and which appear only now and then, and only the server's log of the replies you actually received counts as evidence.

About the assignment

Send the lesson's cases to a real model several times, complete the four checks in the harness, run them on your log, and write up two checks that misfired with the reason.

What you will do

  1. Send the whole case set the number of times the lesson asks, from the console.
  2. Run the harness and read every reply a check failed.
  3. For each misfire, say whether it failed a good reply or passed a bad one, and why.

Common mistakes

  • Blaming the model for a check's mistake

    If a correct reply failed, the check is too strict. That is a finding about the check.

  • Writing notes without reading replies

    The notes are only useful when they point at real replies in your log.

In a real test suite

Teams rerun their checks against the live model on a schedule for exactly this reason: a check suite tuned on recordings drifts out of step with the real model, and misfires are the first sign.

Related lessons

Next lessonRead a trace like a bug reportAll lessons in the courseAI Testing for QA Engineers