Skip to content
CodeKruPro

pass@k, pass^k and severity overrides

46/99

Module · Statistics for testers

intermediate

about 40 min

What this lesson teaches

The idea

When a task is tried several times, "at least one try worked" and "every try worked" are different questions with different answers.

Why a tester cares

A coding helper only needs one good suggestion; an agent handling money must be right every time. Some cases must never fail, whatever the average.

Words you will meet

pass@k:
the chance that at least one of k tries succeeds.
pass^k:
the chance that all k tries succeed.
Hard fail:
a case whose failure blocks the release on its own.

What you will do

  1. 1Implement both metrics
  2. 2Pick one per product
  3. 3Add hard fails to the gate
  4. 4Submit
Jump to practice

When a task is tried several times, two questions have different answers:

  • pass@k: the chance that at least one of k tries succeeds. Right for a coding helper, where a developer can pick the working suggestion.
  • pass^k (pass hat k): the chance that all k tries succeed. Right for an agent that acts on customers' money, where every run must be right.

From c successes in n trials: pass@k = 1 − C(n−c, k) / C(n, k), and pass^k = C(c, k) / C(n, k).

A high average can still hide a case that must never fail. Gates need hard fails: any failure on a case marked severity "hard" blocks the release, whatever the pass rate.

Your task

  1. Implement both metrics in metrics.py.
  2. Choose which metric fits each product.
  3. Change gate.py so a failed hard case blocks the release even when the pass rate is above 90%.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

metrics and gate

How this is checked: When you submit, your script runs on 3 sets of data, 2 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Question

Which metric fits the coding helper?

How this is checked: One answer is right.

3

Question

Which metric fits the refund agent?

How this is checked: One answer is right.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

pass@k vs pass^k for AI agents and coding helpers, with hard-fail severity overrides

AI Testing for QA EngineersIntermediateStatistics for testers

How it works

When a task is tried several times, two questions have different answers. pass@k is the chance that at least one of k tries succeeds, which suits a coding helper where a developer picks the working suggestion. pass^k is the chance that all k tries succeed, which suits an agent acting on customers' money, where every run must be right.

Averages hide cases that must never fail. Release gates add hard fails: any failure on a case marked with hard severity blocks the release, whatever the overall rate.

About the assignment

Implement both metrics from successes and trials, choose the right one for each product, and add a hard-fail rule to the gate.

What you will do

  1. Implement both formulas with combinations, as the page defines them.
  2. Match each product to the metric that reflects how its users experience failure.
  3. Make the gate fail on any hard case failure.

Common mistakes

  • Using pass@k for agents

    An agent that is right once in three tries is wrong for two customers.

  • Letting the average override a hard fail

    A hard case failing blocks the release on its own.

In a real test suite

Agent teams increasingly report pass^k alongside pass@k, because the gap between them shows how consistent an agent is, which matters more to customers than its best run.

Mark hard cases sparingly and on purpose. A list where most cases are hard turns the gate into one that blocks on every wobble, and the team will start overriding it. Reserve the label for money, safety, privacy and legal promises.

Related lessons

Next lessonWrite a binary rubricAll lessons in the courseAI Testing for QA Engineers