Skip to content
CodeKruPro

Re-run your harness in DeepEval

89/99

Module · Frameworks and other languages

advanced

about 45 min

What this lesson teaches

The idea

Frameworks such as DeepEval package what your own harness did: test cases, metrics, thresholds and reports.

Why a tester cares

Knowing both lets you judge what a framework adds and what it hides. Here a scripted judge stands in for a real AI model, so nothing costs money.

Words you will meet

DeepEval:
a Python framework for testing AI outputs with pytest.
Metric:
a function that scores one reply.
GEval:
a DeepEval metric that asks a judge model to score a reply against written criteria.
Threshold:
the lowest score that still passes.

What you will do

  1. 1Write the metric
  2. 2One test per case
  3. 3Add GEval
  4. 4Compare the two
Jump to practice

Frameworks such as DeepEval package what your lesson 15 harness does by hand: test cases, metrics, thresholds and reports. Knowing both lets you judge what a framework adds, and what it hides.

Three DeepEval pieces matter here:

  • LLMTestCase holds the input and the reply.
  • Metrics score a test case. A custom BaseMetric can run any rule, such as your patterns. GEval asks a judge model to score the reply against a written criterion.
  • assert_test fails the pytest test when any metric scores below its threshold.

GEval needs a judge model. This runner has no internet, so judge.py provides a scripted stand-in: it plugs into DeepEval like a real model but scores with fixed rules, which it lists at the top of the file. With a real judge, the same code would call a model, cost money, and sometimes be wrong.

Your task

  1. Finish PatternMetric and turn every case into a test, with GEval on the off-topic case.
  2. Mark what each version of the harness does. On submit your suite runs against a correct bot and three faulty ones.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

against RefundBot

How this is checked: When you submit, your tests run against 6 versions of RefundBot: 2 correct and 4 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

2

Label each one

Which version does this?

0 of 7 labelled

  1. 1Each case shows up as its own test in the pytest report, pass or fail.
  2. 2Prints one pass rate broken down by failure mode.
  3. 3Can score a reply against a written criterion with a judge model (GEval).
  4. 4Checks must_match and must_not_match patterns on every reply.
  5. 5A score can be wrong because the judge model is wrong.
  6. 6Each metric carries its own pass threshold, such as threshold=0.7.
  7. 7Reads its cases from cases.jsonl at run time.

How this is checked: Each row has a right label. All 7 must match. This is a harder check: it earns the second 50 points once the core checks pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

DeepEval tutorial: rebuild an eval harness with a custom metric and GEval

AI Testing for QA EngineersAdvancedFrameworks and other languages

How it works

Frameworks such as DeepEval package what a hand-written harness does: test cases, metrics, thresholds and reports. Knowing both lets you judge what a framework adds and what it hides.

Three pieces matter. An LLMTestCase holds the input and the reply. Metrics score a test case: a custom metric built on DeepEval's base class can run any rule, such as pattern checks, and GEval asks a judge model to score the reply against a written criterion. assert_test fails the pytest test when any metric falls below its threshold.

GEval needs a judge model. The lesson's runner has no internet, so a scripted stand-in judge plugs into DeepEval like a real model but scores with fixed rules listed in its file. With a real judge the same code would call a model, cost money, and sometimes be wrong.

About the assignment

Finish the pattern metric, turn every case into its own test, use GEval with the scripted judge on the off-topic case, and mark what each version of the harness does.

What you will do

  1. Make the metric set its score, success and reason, and return the score.
  2. Parametrize one test over the case file, named by case ID.
  3. Add GEval with a criterion about declining off-topic questions.

Common mistakes

  • A metric that always succeeds

    Set success from the score and threshold, or every test passes.

  • Forgetting must-not-match

    The second pattern is what catches replies that contain the right words and a wrong promise.

In a real test suite

Frameworks earn their place in reporting and in shared metrics across teams. Keeping your own deterministic checks inside them, as custom metrics, keeps costs and judge errors where you can see them.

Related lessons

Next lessonPromptfoo assertions, a CI gate, and moving an evalAll lessons in the courseAI Testing for QA Engineers