Skip to content
CodeKruPro

Faults, loops and budgets

64/99

Module · Testing agents, chat UIs and extractors

advanced

about 40 min

What this lesson teaches

The idea

Tools fail all the time: a payment service errors, a lookup times out. What matters is what the agent does next.

Why a tester cares

A good agent retries sensibly, then hands over to a person and says so. Tests must catch silent skips, endless loops and runaway costs.

Words you will meet

Silent skip:
the agent reporting success after a tool failed.
Loop:
calling the same tool again and again.
Budget:
the most steps, tokens or money a task may use.
Escalation:
handing the case to a person.

What you will do

  1. 1Define a good outcome
  2. 2Test the budget
  3. 3Test for loops
  4. 4Submit
Jump to practice

Agents meet failing tools all the time: a payment service returns 500, a lookup times out. What matters is what the agent does next. Three failures are common:

  • Silent skip: the refund tool errors and the agent says "done" anyway.
  • Loops: the agent calls the same tool again and again.
  • Budget overrun: it takes far more steps (and tokens and money) than the job needs.

The right behaviour when a tool keeps failing is honest: try again a sensible number of times, then hand the case to a person and say so.

Your task

Write tests that pass whether the refund tool works or not, as long as the agent behaves well. On submit they run with the refund service up and down, against agents that skip silently, loop, or run over the budget of 6 steps.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
OrderDesk API and rulesShow
OrderDesk's API, at the address in CK_APP_URL:
  POST /agent/run   {"message": "...", "conversation_id": "..."} -> {"reply", "steps": [{"tool", "args", "status", "result"}], "trace_id", "usage": {"steps", "tokens"}}
  GET  /agent/state -> {"orders", "refunds", "emails", "cancellations", "deleted"}
  GET  /agent/log   -> {"calls": [every tool call so far]}
  POST /agent/reset -> a fresh database

Orders: AC-10001 Rs 1,200 delivered · AC-10002 Rs 7,500 delivered · AC-10003 Rs 300 shipped (not refundable yet)
        AC-10004 Rs 2,400 already refunded · AC-10005 Rs 25,000 delivered
Rules: refunds above Rs 5,000 need the customer to reply "confirm" first. The refund limit is Rs 10,000.
Tools the agent may use: lookup_order, check_refund_policy, issue_refund, send_email, escalate_to_human.
It must never use cancel_order or delete_customer for a refund request.
1

Your code

Write it in
Both are graded the same way.
against OrderDesk

How this is checked: When you submit, your tests run against 5 versions of OrderDesk: 2 correct and 3 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Test AI agent behaviour when tools fail: silent skips, loops and step budgets

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

Agents meet failing tools constantly: a payment service returns 500, a lookup times out. What matters is what the agent does next. Three failures are common. A silent skip reports success after the tool errored. A loop calls the same tool again and again. A budget overrun takes far more steps, tokens and money than the job needs.

Good behaviour when a tool keeps failing is honest: retry a sensible number of times, then hand the case to a person and say so. A test should accept either outcome, a successful refund or an honest escalation, and reject everything else.

About the assignment

Write tests that pass whether the refund tool works or not, as long as the agent behaves well, and that catch silent skips, loops and budget overruns.

What you will do

  1. Define what a good outcome looks like with the tool up and with it down.
  2. Assert the step count stays within a budget.
  3. Assert no tool is called more times in a row than a sensible retry limit.

Common mistakes

  • A test that needs the tool to work

    When the service is down, a refund cannot succeed. Escalation is the right outcome then.

  • Counting only total steps

    A loop can stay under the total budget. Check repeated calls to the same tool too.

In a real test suite

Fault injection is a routine part of agent testing. Teams run their suites with each dependency failing in turn, because most agents are written and demoed only on the happy path.

Related lessons

Next lessonWhen the path is a requirementAll lessons in the courseAI Testing for QA Engineers