Skip to content
CodeKruPro

When the path is a requirement

65/99

Module · Testing agents, chat UIs and extractors

advanced

about 40 min

What this lesson teaches

The idea

Most agent tests should check outcomes, because a good agent may take more than one correct path. But some steps are rules in themselves.

Why a tester cares

Large refunds must wait for the customer to confirm, and some tools must never be used. Those rules get their own tests, next to the outcome checks.

Words you will meet

Outcome test:
a test of the end result, whatever path got there.
Path:
the order of tool calls the agent made.

What you will do

  1. 1Drop the exact sequence
  2. 2Require confirmation
  3. 3Ban tools
  4. 4Submit
Jump to practice

The last two lessons checked outcomes. Most of the time that is right: an agent may reach a correct refund by more than one good path, and a test that demands one exact sequence of tools fails good agents. The starter test does exactly that.

But some steps are requirements, not implementation details:

  • Refunds above Rs 5,000 must wait for the customer to confirm. The state must not change before the confirmation.
  • Some tools must never be used for a refund, whatever the outcome.

Your task

Replace the exact-sequence test with an outcome test, then add tests for the two required behaviours. On submit your tests run against two working agents that take different paths, an agent that skips confirmation, and one that uses a banned tool.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
OrderDesk API and rulesShow
OrderDesk's API, at the address in CK_APP_URL:
  POST /agent/run   {"message": "...", "conversation_id": "..."} -> {"reply", "steps": [{"tool", "args", "status", "result"}], "trace_id", "usage": {"steps", "tokens"}}
  GET  /agent/state -> {"orders", "refunds", "emails", "cancellations", "deleted"}
  GET  /agent/log   -> {"calls": [every tool call so far]}
  POST /agent/reset -> a fresh database

Orders: AC-10001 Rs 1,200 delivered · AC-10002 Rs 7,500 delivered · AC-10003 Rs 300 shipped (not refundable yet)
        AC-10004 Rs 2,400 already refunded · AC-10005 Rs 25,000 delivered
Rules: refunds above Rs 5,000 need the customer to reply "confirm" first. The refund limit is Rs 10,000.
Tools the agent may use: lookup_order, check_refund_policy, issue_refund, send_email, escalate_to_human.
It must never use cancel_order or delete_customer for a refund request.
1

Your code

Write it in
Both are graded the same way.
against OrderDesk

How this is checked: When you submit, your tests run against 4 versions of OrderDesk: 2 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

AI agent test for required confirmation and banned tools, without demanding one exact path

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

Most agent tests should check outcomes, because a good agent may reach a correct refund by more than one path. A test that demands one exact sequence of tool calls fails good agents and teaches nothing.

Some steps, though, are requirements. Refunds above a threshold must wait for the customer's confirmation, and the state must not change before it. Some tools must never be used for a refund, whatever the outcome. Those rules are tested directly, alongside outcome checks.

About the assignment

Replace the exact-sequence test with an outcome test, then add tests for confirmation before large refunds and for tools that are never allowed.

What you will do

  1. Delete the assertion on the exact list of tools.
  2. For a large refund, assert no money moves until the customer confirms, and that it moves after.
  3. Assert no forbidden tool appears in any step.

Common mistakes

  • Checking the reply for a question

    An agent can ask for confirmation in words and still issue the refund. Check the state.

  • Keeping the exact sequence

    Two working agents take different paths. A path test fails one of them.

In a real test suite

Required steps and banned tools are where permission boundaries meet tests. Teams usually enforce them in the tool layer as well, and test both, so a prompt change cannot silently remove them.

Name each required behaviour after the rule it protects, such as confirmation before large refunds. When a test fails months later, the name tells the reader which business rule broke, not just which assertion did.

Related lessons

Next lessonMulti-turn: first failure and N-1 replayAll lessons in the courseAI Testing for QA Engineers