Skip to content
CodeKruPro

Property checks: facts, forbidden content, length

12/99

Module · A deterministic harness

beginner

about 35 min

What this lesson teaches

The idea

When you cannot predict the exact words, test properties: a required fact appears, a forbidden promise does not, the reply stays short enough, and it is in the right language.

Why a tester cares

Properties are what keep AI tests stable. The skill is precision: a check that rejects any other number also fails a correct reply that mentions a competitor.

Words you will meet

Property check:
a test of something true of every correct reply.
Forbidden content:
text that must never appear, such as a promise the policy does not allow.

What you will do

  1. 1Read the six docstrings
  2. 2Write each check
  3. 3Submit
Jump to practice

When you cannot predict the exact words, you test properties: things every correct reply must be true of, however it is phrased. The editor has six empty tests, each with a sentence saying what it checks. Each one asks RefundBot the window question three times.

Your task

Fill in all six checks. On submit they run against three correct builds that word things differently, and against faulty ones.

One correct build compares Acme with other shops, for example by mentioning the shorter window another store offers. Your checks must work out which number belongs to Acme. A check that rejects every reply containing another number would fail a correct bot.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

1

Your code

against the practice bot

How this is checked: When you submit, your tests run against 5 versions of RefundBot: 3 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Property based checks for LLM replies: right facts, forbidden content and sensible length

AI Testing for QA EngineersBeginnerA deterministic harness

How it works

When the exact words cannot be predicted, test properties: statements that are true of every correct reply. Typical properties are that a required fact appears, that a forbidden promise does not, that the reply stays within a length budget, and that it answers in the customer's language.

The subtle part is precision. A correct reply may mention other numbers, such as a competitor's shorter window, so a check that rejects any other number fails a good bot. A good property identifies which number belongs to Acme.

About the assignment

Fill in six property checks described by their docstrings, so they pass on three differently worded correct builds and fail on the faulty ones.

What you will do

  1. Read every docstring before writing any check.
  2. Ask the bot several times inside each check, as the template does.
  3. Make each check fail only for the defect it names.

Common mistakes

  • Over-strict checks

    Rejecting every reply that mentions another number fails a correct bot that compares stores.

  • Checking only the first reply

    Each check asks more than once. Every reply must hold the property.

In a real test suite

A suite of sharp properties is usually the cheapest layer of an AI test strategy. Teams add LLM judges only for qualities such as tone or relevance that properties cannot decide.

Length checks need care too. A budget in characters or sentences catches rambling replies, but set it from correct replies you have seen, not from a guess, or a helpful answer that adds one true detail will fail.