Skip to content
CodeKruPro

Note what went wrong, and where it first went wrong

19/99

Module · Read before you measure

beginner

about 40 min

What this lesson teaches

The idea

Error analysis means reading real traces one at a time, marking each pass or fail, and noting the first step that went wrong.

Why a tester cares

Later steps often look broken only because an earlier step fed them bad input. Fixing the first failure fixes the whole chain.

Words you will meet

Error analysis:
reading real failures to learn what actually goes wrong, before measuring anything.
First failing step:
the earliest step in a trace that did the wrong thing.

What you will do

  1. 1Read each trace
  2. 2Pass or fail
  3. 3First failing step
  4. 4Submit
Jump to practice

Error analysis starts with a simple habit: read real traces one at a time, write down pass or fail, and for each failure mark the first step that went wrong. Later steps often look broken only because an earlier one was. Fixing the first failure fixes the chain.

The steps a request goes through:

  • Understanding: the bot misread what was asked (its plan shows it).
  • Retrieval: the wrong, outdated or forbidden document came back.
  • Tool call: the wrong tool, or the right tool with wrong arguments.
  • Using the result: a tool or document said one thing and the bot did another.
  • Final answer: everything before was right, and the reply still went wrong.

Your task

Label all 20 traces pass or fail. Then, for each one, choose the first failing step, or "Nothing failed" for a pass.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account

Twenty traces

T01Return window, 12 days after deliveryRefundBot
  1. 1. CustomerI got my kettle 12 days ago. Can I still send it back?
  2. 2. Policy foundMost items can be returned within 30 days of delivery for a full refund.
  3. 3. ReplyYes. At 12 days you are inside our 30-day return window, so you can send it back.
T02Gift card refundRefundBot
  1. 1. CustomerCan I get my money back for a gift card I bought?
  2. 2. Policy foundGift cards cannot be refunded.
  3. 3. ReplyYes, gift cards can be refunded within 30 days of purchase.
T03Annual leave daysPolicyPal
  1. 1. EmployeeHow many days of annual leave do I get?
  2. 2. Documents ranked1. leave-2023 (replaced in 2025): 18 days a year 2. leave-2025: 24 days a year
  3. 3. ReplyFull-time employees get 18 days of paid annual leave a year. [leave-2023]
T04Refund for AC-10001OrderDesk
  1. 1. CustomerPlease refund order AC-10001.
  2. 2. lookup_order(AC-10001){"status": "delivered", "amount": 1200, "refunded": 0}
  3. 3. issue_refund(AC-10001, 1200){"refund_id": "RF-0001"}
  4. 4. ReplyDone. I have refunded Rs 1,200 for order AC-10001 and emailed you a confirmation.
T05Partial refundOrderDesk
  1. 1. CustomerOnly the lid was missing. Please refund Rs 200 on AC-10001.
  2. 2. Agent planCustomer wants a refund for AC-10001. Refund the full order.
  3. 3. issue_refund(AC-10001, 1200){"refund_id": "RF-0002"}
  4. 4. ReplyDone. Rs 1,200 has been refunded for AC-10001.
T06Order statusOrderDesk
  1. 1. CustomerWhere is AC-10003?
  2. 2. lookup_order(AC-10003){"status": "shipped", "expected": "Friday"}
  3. 3. ReplyYour order AC-10003 was delivered yesterday.
T07Damaged itemRefundBot
  1. 1. CustomerMy blender arrived cracked.
  2. 2. Policy foundDamaged items: report within 48 hours of delivery with a photo; pickup is free.
  3. 3. ReplySorry about that. Send us a photo within 48 hours of delivery and we will pick it up free.
T08Salary bands asked by a sales employeePolicyPal
  1. 1. Employee (Sales)What are the engineering salary bands?
  2. 2. Documents ranked1. eng-bands-2025 (Engineering managers only)
  3. 3. ReplyEngineering salary bands: L3 Rs 18 to 24 lakh, L4 Rs 26 to 34 lakh. [eng-bands-2025]
T09Hinglish window questionRefundBot
  1. 1. Customerreturn kitne din tak kar sakte hain?
  2. 2. Policy foundMost items can be returned within 30 days of delivery for a full refund.
  3. 3. ReplyAap 14 din tak return kar sakte hain.
T10Refund service downOrderDesk
  1. 1. CustomerRefund AC-10001 please.
  2. 2. lookup_order(AC-10001){"status": "delivered", "amount": 1200}
  3. 3. issue_refund(AC-10001, 1200)HTTP 500: refund service unavailable
  4. 4. ReplyAll set: Rs 1,200 is refunded for AC-10001.
T11Work from home daysPolicyPal
  1. 1. EmployeeHow many days a week can I work from home?
  2. 2. Documents ranked1. wfh-2025: up to 2 days a week with manager approval
  3. 3. ReplyYou may work from home up to 2 days a week with your manager's approval. [wfh-2025]
T12Off-topic questionRefundBot
  1. 1. CustomerWho will win the cricket match today?
  2. 2. ReplyI can only help with returns, refunds and orders at Acme.
T13Refund on the wrong orderOrderDesk
  1. 1. CustomerRefund AC-10001.
  2. 2. lookup_order(AC-10001){"status": "delivered", "amount": 1200}
  3. 3. issue_refund(AC-10002, 1200){"refund_id": "RF-0003"}
  4. 4. ReplyDone. Rs 1,200 refunded for AC-10001.
T14Dress codePolicyPal
  1. 1. EmployeeWhat is the dress code?
  2. 2. Documents ranked(no matching document)
  3. 3. ReplyThe dress code is business formal from Monday to Thursday. [health-2025]
T15Pickup feeRefundBot
  1. 1. CustomerDo you charge for pickup?
  2. 2. Policy foundOther return pickups cost Rs 99; damaged items are picked up free.
  3. 3. ReplyA return pickup costs Rs 99, and it is free for damaged items.
T16Large refund without confirmationOrderDesk
  1. 1. CustomerRefund AC-10002.
  2. 2. lookup_order(AC-10002){"status": "delivered", "amount": 7500}
  3. 3. issue_refund(AC-10002, 7500){"refund_id": "RF-0004"}
  4. 4. ReplyDone. Rs 7,500 refunded.
T17Notice periodPolicyPal
  1. 1. EmployeeWhat is my notice period?
  2. 2. Documents ranked1. notice-2025: 60 days, or 30 during probation
  3. 3. ReplyI couldn't find that in Acme's HR policies. Please ask HR.
T18Refund timeRefundBot
  1. 1. CustomerWhen will I get my money back?
  2. 2. Policy foundRefunds reach the original payment method within 5 to 7 business days.
  3. 3. ReplyYour refund reaches your original payment method within 5 to 7 business days of us receiving the item.
T19Status for an unknown orderOrderDesk
  1. 1. CustomerWhere is AC-99999?
  2. 2. lookup_order(AC-99999){"error": "No order AC-99999"}
  3. 3. ReplyI could not find order AC-99999. Please check the order ID.
T20Travel bookingPolicyPal
  1. 1. EmployeeHow early must I book business travel?
  2. 2. Documents ranked1. travel-2025: book at least 7 days ahead
  3. 3. ReplyBook business travel at least 7 days ahead. [expenses-2025]
1

Label each one

Pass or fail

0 of 20 labelled

  1. 1T01: Return window, 12 days after delivery
  2. 2T02: Gift card refund
  3. 3T03: Annual leave days
  4. 4T04: Refund for AC-10001
  5. 5T05: Partial refund
  6. 6T06: Order status
  7. 7T07: Damaged item
  8. 8T08: Salary bands asked by a sales employee
  9. 9T09: Hinglish window question
  10. 10T10: Refund service down
  11. 11T11: Work from home days
  12. 12T12: Off-topic question
  13. 13T13: Refund on the wrong order
  14. 14T14: Dress code
  15. 15T15: Pickup fee
  16. 16T16: Large refund without confirmation
  17. 17T17: Notice period
  18. 18T18: Refund time
  19. 19T19: Status for an unknown order
  20. 20T20: Travel booking

How this is checked: Each row has a right label. At least 16 of 20 must match.

2

Label each one

First failing step

0 of 20 labelled

  1. 1T01: Return window, 12 days after delivery
  2. 2T02: Gift card refund
  3. 3T03: Annual leave days
  4. 4T04: Refund for AC-10001
  5. 5T05: Partial refund
  6. 6T06: Order status
  7. 7T07: Damaged item
  8. 8T08: Salary bands asked by a sales employee
  9. 9T09: Hinglish window question
  10. 10T10: Refund service down
  11. 11T11: Work from home days
  12. 12T12: Off-topic question
  13. 13T13: Refund on the wrong order
  14. 14T14: Dress code
  15. 15T15: Pickup fee
  16. 16T16: Large refund without confirmation
  17. 17T17: Notice period
  18. 18T18: Refund time
  19. 19T19: Status for an unknown order
  20. 20T20: Travel booking

How this is checked: Each row has a right label. At least 7 of 10 key rows must be right. This is a harder check: it earns the second 50 points once the core checks pass.

3

Write it up

Write a short note for each failure in your own words. You will group these notes in the next lesson.

How this is checked: Not graded. It is there to help you think.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

LLM error analysis: marking the first failing step in a trace

AI Testing for QA EngineersBeginnerRead before you measure

How it works

Error analysis starts with a plain habit: read real traces one at a time, mark pass or fail, and for each failure note the first step that went wrong. Later steps often look broken only because an earlier one fed them bad input, so fixing the first failure fixes the chain.

The steps of a request are usually understanding, retrieval, tool calls, using the result, and the final answer. Naming the first failing step turns a vague "the bot was wrong" into a fixable claim about one component.

About the assignment

Label twenty traces pass or fail, choose the first failing step for each failure, and write a short note.

What you will do

  1. Decide pass or fail from the final reply and the policy.
  2. For each failure, walk the steps from the start and stop at the first wrong one.
  3. Choose "nothing failed" for every pass.

Common mistakes

  • Blaming the last step

    A wrong final answer is often caused by a wrong retrieval or tool argument earlier.

  • Marking passes with a step

    A trace that passed has no failing step.

In a real test suite

Teams run this exercise weekly on a sample of production traces. The counts per first-failing step tell them where to invest: better retrieval, stricter tool arguments, or prompt work.

Aim for consistency over speed. When two traces fail the same way, they should get the same first failing step; if your labels drift halfway through, re-read the early traces before moving on.

Related lessons

Next lessonTurn notes into a failure taxonomyAll lessons in the courseAI Testing for QA Engineers