Skip to content
CodeKruPro

pytest parametrize, fixtures and markers for evals

10/99

Module · Python and LLM APIs bridge

beginner

about 25 min

What this lesson teaches

The idea

Keep test cases in a data file, one per line, and let pytest run one test function once for each row.

Why a tester cares

Anyone can add a case, often one per reported bug, without writing new code, and a failure names the exact case that broke.

Words you will meet

pytest:
the most common Python test runner.
Parametrize:
run one test once per row of data.
Fixture:
setup code, such as creating an API client, that pytest hands to every test that asks for it.
JSONL:
a file with one JSON object on each line.

What you will do

  1. 1Read the data file
  2. 2One parametrized test
  3. 3Submit
Jump to practice

An eval suite grows one case at a time: every bug a customer reports becomes a new row. If each row needs a new test function, nobody adds rows. The usual fix is to keep cases in a data file and drive one test from it.

cases.jsonl holds one JSON object per line: an id, a question, phrases the reply must_contain, and phrases it must_not_contain.

Your task

  1. Replace the three hand-written tests with one test that pytest runs once per row of cases.jsonl (@pytest.mark.parametrize), using each row's id as the test id.
  2. Give the test the API client through a pytest fixture called client.
  3. Check every phrase in both lists, ignoring upper and lower case.

On submit, the file may contain rows you have not seen. Your test must pick them up without any change to the code.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

against the practice bot

How this is checked: When you submit, your tests run against 4 versions of RefundBot: 2 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

pytest parametrize eval cases from JSONL with fixtures, so new cases need no new code

AI Testing for QA EngineersBeginnerPython and LLM APIs bridge

How it works

An eval suite grows one case at a time, often one per reported bug. If every case needs a new test function, people stop adding them. Keeping cases in a data file, one JSON object per line, lets anyone add a row without touching code.

pytest supports this directly. @pytest.mark.parametrize runs one test function once per data row, and the ids argument names each run after the row, so a failure tells you which case broke. A fixture builds the API client once and hands it to every test that asks for it by name.

About the assignment

Replace three hand-written tests with one parametrized test driven by cases.jsonl, named by each row's id, receiving the client from a fixture, and checking every phrase each row lists.

What you will do

  1. Read the data file at import time into a list of dicts.
  2. Parametrize one test over that list and set readable ids.
  3. Check both phrase lists for each row, ignoring case.

Common mistakes

  • Hard-coding the rows

    Grading adds rows you have not seen. Read the file when the tests run.

  • Checking only the first phrase

    A row can list several phrases. Every one of them must hold.

In a real test suite

Data-driven suites are how eval sets become team assets. Product owners and support staff can propose cases as rows, and the same test code covers them, which keeps the suite growing with real customer problems.

Related lessons

Next lessonSchema tests for structured outputAll lessons in the courseAI Testing for QA Engineers