Skip to content
CodeKruPro

Tool choice and arguments

63/99

Module · Testing agents, chat UIs and extractors

advanced

about 35 min

What this lesson teaches

The idea

Agents fail in their tool calls more than anywhere else: they pick the wrong tool, or the right tool with the wrong values.

Why a tester cares

The worst case succeeds quietly, such as refunding the wrong customer's order. Checking the tool, its arguments and the result separately catches each kind.

Words you will meet

Tool call:
the agent asking the app to run a function, such as refund(order, amount).
Argument:
a value passed to a tool, such as the amount.

What you will do

  1. 1Read the steps
  2. 2Test the tool
  3. 3Test the arguments
  4. 4Submit
Jump to practice

Agents fail in their tool calls more than anywhere else, and in two separate ways:

  • Tool choice: the wrong tool, such as cancelling an order when asked for a refund.
  • Arguments: the right tool with the wrong values: the full amount instead of the part asked for, or another customer's order.

The worst case is a wrong argument that still succeeds. Refunding the wrong order raises no error, and the agent reports success. Every step OrderDesk took is in the steps of its response: tool, arguments, status and result.

Your task

Ask for a partial refund and test the tool choice and the arguments separately, plus the database. On submit your tests run against a working build and builds that choose the wrong tool, the wrong amount and the wrong order.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
OrderDesk API and rulesShow
OrderDesk's API, at the address in CK_APP_URL:
  POST /agent/run   {"message": "...", "conversation_id": "..."} -> {"reply", "steps": [{"tool", "args", "status", "result"}], "trace_id", "usage": {"steps", "tokens"}}
  GET  /agent/state -> {"orders", "refunds", "emails", "cancellations", "deleted"}
  GET  /agent/log   -> {"calls": [every tool call so far]}
  POST /agent/reset -> a fresh database

Orders: AC-10001 Rs 1,200 delivered · AC-10002 Rs 7,500 delivered · AC-10003 Rs 300 shipped (not refundable yet)
        AC-10004 Rs 2,400 already refunded · AC-10005 Rs 25,000 delivered
Rules: refunds above Rs 5,000 need the customer to reply "confirm" first. The refund limit is Rs 10,000.
Tools the agent may use: lookup_order, check_refund_policy, issue_refund, send_email, escalate_to_human.
It must never use cancel_order or delete_customer for a refund request.
1

Your code

Write it in
Both are graded the same way.
against OrderDesk

How this is checked: When you submit, your tests run against 4 versions of OrderDesk: 1 correct and 3 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Test AI agent tool calls and arguments: wrong tool, wrong amount, wrong order

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

Agents fail in their tool calls more than anywhere else, in two separate ways. Tool choice goes wrong when the agent picks the wrong tool, such as cancelling an order when asked for a refund. Arguments go wrong when the right tool gets the wrong values: the full amount instead of the part asked for, or another customer's order.

The worst case is a wrong argument that still succeeds. Refunding the wrong order raises no error, and the agent reports success. OrderDesk returns every step it took, with tool, arguments, status and result, so a test can check each part on its own.

About the assignment

Ask OrderDesk for a partial refund and test the tool it chose, the arguments it used and the resulting database state as separate checks.

What you will do

  1. Read the steps in a response to see how tool calls are recorded.
  2. Assert the refund tool was used and no forbidden tool appeared.
  3. Assert the order ID and amount in the arguments, then check the state as well.

Common mistakes

  • One combined assertion

    Separate checks tell a developer which kind of failure happened.

  • Trusting the step status

    A wrong-order refund has status ok. Compare the arguments with what was asked.

In a real test suite

Tool-call checks catch regressions in prompts and tool descriptions quickly, because a changed description often shows up first as a different tool choice before any outcome changes.

Logging tool arguments in production, with personal data masked, lets teams sample and test real tool calls the same way.

Related lessons

Next lessonFaults, loops and budgetsAll lessons in the courseAI Testing for QA Engineers