Skip to content
CodeKruPro

Cold Product Test

99/99

Module · Career, capstones and the Cold Product Test

advanced

about 240 min

What this lesson teaches

The idea

The Cold Product Test is the final test: an unfamiliar AI product with a spec, an API and a chat window, and no hints. You produce evidence a release owner could act on.

Why a tester cares

It is shaped like the take-home tasks employers set. Your copy of TripMate has its own numbers and defects, so the work is yours alone.

Words you will meet

Cold test:
testing a product you have never seen, without guidance.
Spec:
the written description of how the product should behave.
Release owner:
the person who decides whether the product ships.

What you will do

  1. 1Read your spec
  2. 2Plan
  3. 3Test
  4. 4Hunt and report
  5. 5Compute
  6. 6Decide
Jump to practice

This is the course's final test, and the definition of job-ready it was built around: given an unfamiliar AI product with a spec, an API and a chat window, produce evidence a release owner could act on, without hints.

The product is TripMate, a bus-ticket support assistant. It is generated for you: your spec's numbers and the defects planted in your console build are your own, so your answers will not match anyone else's. Read your spec first: type /spec in the console, or GET /trip/spec.

Deliver six things:

  1. An approach: the risks first, what you will test in order, and what you will not test.
  2. A taxonomy of the ways TripMate can fail.
  3. A test suite. On submit it runs against correct builds and six single-defect builds of your product.
  4. Defect reports from your console build, each citing a trace. Report nothing that works.
  5. Two numbers from your spec.
  6. A decision and a memo.

The Cold Product Test is practice evidence, not proctored proof. Set aside four hours, and treat it as you would a take-home assignment from an employer.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
TripMate APIShow
TripMate's API, at the address in CK_APP_URL. Send the header X-Customer: cust-a on every call.
  GET  /trip/spec                -> {"text": your product's spec}
  GET  /trip/bookings/<id>       -> the booking, 403 if it belongs to another customer
  POST /trip/quote {"booking_id": "TM-1003"} -> {"booking_id", "hours_left", "fare", "fee", "refund"}
  POST /trip/ask {"question": "..."} -> {"answer"}
In the console, type /spec to read your product's spec. Bookings are listed at the end of the spec.
1

Your TripMate (you are customer cust-a)

No messages yet. Start with a suggested question below.

2

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. Compare it with the model answer after you submit.

3

Write it up

How this is checked: When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks. Compare it with the model answer after you submit.

4

Your code

against your TripMate

How this is checked: When you submit, your tests run against 8 versions of TripMate: 2 correct and 6 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

5

Write it up

How this is checked: When you submit, each report is matched with the faults planted in your copy of the product, and each must cite a trace ID from your own log. Reports that match nothing count against you.

6

Your answer

How this is checked: Your copy of the product has its own numbers, so the right value is worked out from your copy.

7

Your answer

How this is checked: Your copy of the product has its own numbers, so the right value is worked out from your copy. This is a harder check: it earns the second 50 points once the core checks pass.

8

Question

Your release decision for your console build

How this is checked: One answer is right.

9

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass. Compare it with the model answer after you submit.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Take home test for AI QA engineers: the Cold Product Test on an unseen AI product

AI Testing for QA EngineersAdvancedCareer, capstones and the Cold Product Test

How it works

The Cold Product Test is the course's final test and its definition of job-ready: given an unfamiliar AI product with a spec, an API and a chat window, produce evidence a release owner could act on, without hints. It is practice evidence rather than proctored proof, and it is shaped like the take-home assignments employers set.

The product, TripMate, is generated for each learner. Your spec's numbers and the defects planted in your console build are your own, so answers cannot be shared. Your test suite runs against correct builds and single-defect builds of the same product, which share your spec.

About the assignment

Plan, build a taxonomy, write a suite, report defects from your console build with traces, compute two numbers from your spec, and decide with a memo.

What you will do

  1. Read your spec before anything else.
  2. Write the approach with risks first and explicit exclusions.
  3. Make your suite read the spec at run time instead of hard-coding its numbers.
  4. Report only defects you reproduced, and work the numbers out from your own spec.

Common mistakes

  • Hard-coding numbers

    Your suite's expected values should come from the spec the API serves.

  • Reporting defects you did not see

    Defects not planted in your build count as false reports.

In a real test suite

Treat it as a real take-home: set aside the time, work without help, and keep your notes. The artefacts belong in your portfolio, and walking an interviewer through how you approached an unseen product is one of the strongest answers you can give.

Related lessons

All lessons in the courseAI Testing for QA Engineers