Skip to content
CodeKruPro

Multi-turn: first failure and N-1 replay

66/99

Module · Testing agents, chat UIs and extractors

advanced

about 35 min

What this lesson teaches

The idea

A failure in a long conversation depends on everything said before it, which makes it hard to reproduce.

Why a tester cares

Find the first turn that went wrong, then replay only the turns before it and send that one. You get a short, fast regression test instead of a long fragile script.

Words you will meet

Turn:
one message and its reply in a conversation.
Multi-turn:
a conversation with several turns.
N-1 replay:
replaying every turn before the failing one, then sending the failing turn.

What you will do

  1. 1Read the conversation
  2. 2Find the first failure
  3. 3Replay N-1 turns
  4. 4Submit
Jump to practice

Conversation failures are hard to test because they depend on everything said before. The trick is the same as in lesson 19: find the first turn that went wrong. Everything after it may only be fallout.

Then reproduce it in its smallest form: replay the turns before the failure (N−1 turns) in one conversation, send the failing turn, and assert what should have happened. That gives a short, fast regression test instead of a long scripted chat.

Your task

  1. Read the customer's conversation and name the first turn where OrderDesk failed.
  2. Write a test that replays only the turns before it, sends the failing turn, and checks the outcome. On submit it runs on a working agent and on the build that loses context.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
OrderDesk API and rulesShow
OrderDesk's API, at the address in CK_APP_URL:
  POST /agent/run   {"message": "...", "conversation_id": "..."} -> {"reply", "steps": [{"tool", "args", "status", "result"}], "trace_id", "usage": {"steps", "tokens"}}
  GET  /agent/state -> {"orders", "refunds", "emails", "cancellations", "deleted"}
  GET  /agent/log   -> {"calls": [every tool call so far]}
  POST /agent/reset -> a fresh database

Orders: AC-10001 Rs 1,200 delivered · AC-10002 Rs 7,500 delivered · AC-10003 Rs 300 shipped (not refundable yet)
        AC-10004 Rs 2,400 already refunded · AC-10005 Rs 25,000 delivered
Rules: refunds above Rs 5,000 need the customer to reply "confirm" first. The refund limit is Rs 10,000.
Tools the agent may use: lookup_order, check_refund_policy, issue_refund, send_email, escalate_to_human.
It must never use cancel_order or delete_customer for a refund request.

A customer's conversation

conv-17Refund for a broken lidOrderDesk
  1. 1. CustomerHi, I need help with order AC-10001.
  2. 2. OrderDeskThanks, I have order AC-10001. Do you want a refund or a status update?
  3. 3. CustomerIt arrived with a broken lid.
  4. 4. OrderDeskSorry to hear that. Do you want a refund for order AC-10001?
  5. 5. CustomerYes, please refund it.
  6. 6. OrderDeskWhich order is this about? It looks like AC-12345.
  7. 7. CustomerAC-10001, I told you already.
  8. 8. OrderDeskThanks, I have order AC-10001. Do you want a refund or a status update?
  9. 9. CustomerA REFUND. Please.
  10. 10. OrderDeskWhich order is this about? It looks like AC-12345.
1

Your answer

How this is checked: Your number must match the right value.

2

Your code

Write it in
Both are graded the same way.
against OrderDesk

How this is checked: When you submit, your tests run against 2 versions of OrderDesk: 1 correct and 1 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Multi turn chatbot regression test with N-1 replay of the turns before the failure

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

Conversation failures depend on everything said before, which makes them hard to reproduce. The approach from trace analysis applies: find the first turn that went wrong, because everything after it may only be fallout.

Then reproduce it in its smallest form. Replay the turns before the failure in one conversation, send the failing turn, and assert what should have happened. That gives a short, fast regression test instead of a long scripted chat that breaks whenever a harmless earlier turn changes.

About the assignment

Name the first turn where OrderDesk failed in a customer conversation, and write a test that replays only the turns before it, sends that turn, and checks the outcome.

What you will do

  1. Read the conversation and find the first reply that was wrong.
  2. Replay only the earlier turns, in one conversation ID.
  3. Send the failing turn and assert the outcome the customer should have got.

Common mistakes

  • Replaying the whole conversation

    Extra turns after the failure add time and make the test fragile without adding coverage.

  • Separate conversation IDs

    The failure depends on context. All replayed turns must share one conversation.

In a real test suite

Teams build libraries of these minimal replays from production incidents. Each one is a few lines long and runs in seconds, which keeps multi-turn coverage affordable.

The same idea works for agents with memory: replay what the agent knew just before the mistake, then test the step that went wrong.

Related lessons

Next lessonMCP contract test: pin and diff toolsAll lessons in the courseAI Testing for QA Engineers