Skip to content
CodeKruPro

Grade the outcome, not the claim

62/99

Module · Testing agents, chat UIs and extractors

advanced

about 35 min

What this lesson teaches

The idea

An agent's reply is only a claim. "Your refund is done" is what a working agent says, and also what a broken one says when the refund never happened.

Why a tester cares

Agent tests check the world, not the words: after the agent acts, read the database it was supposed to change.

Words you will meet

Agent:
an AI that takes actions through tools, here refunds in an order system.
State:
the data the agent changes, such as an order's balance.
Fixture:
setup code that resets the data before each test.

What you will do

  1. 1Read the API
  2. 2Check the database
  3. 3Cover partial and refused refunds
  4. 4Submit
Jump to practice

An agent's reply is a claim. "Your refund is done" is what a working agent says, and also what a broken one says when its refund tool never ran. The first rule of agent testing: check the world, not the words. After the agent acts, read the state it was supposed to change.

OrderDesk exposes its database through /agent/state. Every test starts from the same five orders.

Your task

Replace the starter's reply check with checks on the database: a full refund, a partial refund, and an order that must not be refunded. On submit your tests run against working builds, which word their replies differently, and a build that claims refunds it never made.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
OrderDesk API and rulesShow
OrderDesk's API, at the address in CK_APP_URL:
  POST /agent/run   {"message": "...", "conversation_id": "..."} -> {"reply", "steps": [{"tool", "args", "status", "result"}], "trace_id", "usage": {"steps", "tokens"}}
  GET  /agent/state -> {"orders", "refunds", "emails", "cancellations", "deleted"}
  GET  /agent/log   -> {"calls": [every tool call so far]}
  POST /agent/reset -> a fresh database

Orders: AC-10001 Rs 1,200 delivered · AC-10002 Rs 7,500 delivered · AC-10003 Rs 300 shipped (not refundable yet)
        AC-10004 Rs 2,400 already refunded · AC-10005 Rs 25,000 delivered
Rules: refunds above Rs 5,000 need the customer to reply "confirm" first. The refund limit is Rs 10,000.
Tools the agent may use: lookup_order, check_refund_policy, issue_refund, send_email, escalate_to_human.
It must never use cancel_order or delete_customer for a refund request.
1

Your code

Write it in
Both are graded the same way.
against OrderDesk

How this is checked: When you submit, your tests run against 3 versions of OrderDesk: 2 correct and 1 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Test AI agent outcomes by checking the database state, not the agent's reply

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

An agent's reply is a claim. "Your refund is done" is what a working agent says, and also what a broken one says when its refund tool never ran. The first rule of agent testing is to check the world, not the words: after the agent acts, read the state it was supposed to change.

OrderDesk exposes its database through a state endpoint, and every test starts from the same five orders thanks to a reset call in a fixture. That makes each test independent and its expected state easy to work out from the order list.

A useful outcome suite covers the full case, a partial case and a refused case, because agents fail differently on each.

About the assignment

Replace the starter's reply check with checks on the database for a full refund, a partial refund and an order that must not be refunded.

What you will do

  1. Read the API panel and the five starting orders.
  2. After each request, read the state and assert the refunds list and order balances.
  3. Add a test for an order that must not be refunded, and assert nothing changed.

Common mistakes

  • Checking the reply text

    A build that claims refunds it never made passes reply checks. Read the state.

  • Sharing state between tests

    Without the reset fixture, one test's refund changes the next test's starting point.

In a real test suite

Outcome checks are the backbone of agent test suites in production. They survive wording changes, different tool paths and model upgrades, because they only care whether the job was done.

Related lessons

Next lessonTool choice and argumentsAll lessons in the courseAI Testing for QA Engineers