Skip to content
CodeKruPro

Schema tests for structured output

11/99

Module · A deterministic harness

beginner

about 30 min

What this lesson teaches

The idea

When a model returns data as JSON, other code trusts it. A schema states exactly which fields must be there, their types and their allowed values.

Why a tester cares

Checking only that a reply is valid JSON misses a missing field or a made-up category. Validating against a schema catches each one and says which rule broke.

Words you will meet

Structured output:
a reply in a fixed data format instead of free text.
JSON Schema:
a description of what a valid JSON object must look like.
Enum:
a field that may only take one of a listed set of values.

What you will do

  1. 1Run the starter
  2. 2Write the schema
  3. 3Validate every reply
  4. 4Submit
Jump to practice

Not every AI feature chats. Acme uses the same model to read customer messages and pull out order details as JSON, which other code then trusts. If the model leaves out a key, invents a reason, or writes a number as text, that downstream code breaks.

The starter test only checks that each reply is valid JSON. That catches a reply cut off halfway, but nothing else.

Your task

Write a JSON Schema for the extraction and validate every reply against it with the jsonschema library. The contract the extractor must honour:

  • exactly these keys: order_id, reason, item, days_since_delivery, refund_eligible, and no others;
  • order_id: text like AC-12345, or null when the message has none;
  • reason: one of damaged, wrong_item, changed_mind, late_delivery, other;
  • item: text or null;
  • days_since_delivery: a whole number of 0 or more, or null;
  • refund_eligible: true or false.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

against the practice bot

How this is checked: When you submit, your tests run against 6 versions of RefundBot: 2 correct and 4 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

JSON Schema validation for LLM structured output: catching missing keys and wrong types

AI Testing for QA EngineersBeginnerA deterministic harness

How it works

When a model extracts data as JSON, other code trusts it. A missing key, an invented category or a number sent as text can break that code silently. Checking that a reply parses as JSON catches only replies cut off halfway.

JSON Schema states the contract precisely: which keys must be present, which are allowed, each value's type, which values an enumerated field may take, and where null is acceptable. The jsonschema library validates a reply against it and reports exactly which rule failed.

About the assignment

Write a schema for the order-extraction contract and validate every reply against it, so the tests fail on each way the extractor can break it.

What you will do

  1. Run the starter to see what checking only valid JSON misses.
  2. Write the schema from the contract on the page, key by key.
  3. Forbid keys the contract does not list, and allow null only where it says so.

Common mistakes

  • Leaving extra keys allowed

    By default a schema accepts keys it does not mention. Turn that off, or an invented field passes.

  • Typing a count as number

    A day count must be a whole number. A schema type of number would accept fractions.

In a real test suite

Schemas double as documentation: the team that consumes the extractor and the team that tests it read the same file. Many providers can also enforce a schema at generation time, but tests still check it, because enforcement is a feature that can be switched off.

Related lessons

Next lessonProperty checks: facts, forbidden content, lengthAll lessons in the courseAI Testing for QA Engineers