Skip to content
CodeKruPro

A model upgrade: what blocks it

82/99

Module · Release, production and operations

advanced

about 40 min

What this lesson teaches

The idea

A newer model often scores higher overall and still breaks something: one language gets worse, or JSON comes back wrapped in extra text.

Why a tester cares

An upgrade is judged on its parts, not its total, by rules agreed in advance. It is also a chance to remove prompt workarounds the old model needed.

Words you will meet

Model upgrade:
switching the product to a newer AI model.
Segment:
one group of cases, such as Hindi questions or JSON replies.
Hard gate:
a case that blocks the release if it fails.

What you will do

  1. 1Score each segment
  2. 2Count format errors
  3. 3Check hard-gate cases
  4. 4Decide
Jump to practice

A newer model often scores higher overall and still breaks something: one language gets worse, JSON comes back wrapped in a code fence, or a case that must never fail starts failing. Judge an upgrade on the parts, not the total.

The team's upgrade rules. Block when any of these is true; otherwise ship:

  1. A hard-gate case fails on the new model.
  2. The new model returns more invalid JSON than the old one in the json segment.
  3. Any segment drops by more than 0.10.

Upgrades are also a chance to simplify. Teams add instructions to work around an old model's weak spots. Test the new model with the shorter prompt too: if it scores no more than 0.02 below the full prompt, the extra instructions can go.

Your task

Score the recorded outputs (models and dates are labelled in the files) and print the figures and the decision. On submit it also runs on three upgrades you have not seen.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on the recorded outputs

How this is checked: When you submit, your script runs on 4 sets of data, 3 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Question

Which rule decides this upgrade?

How this is checked: One answer is right.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Evaluate an LLM model upgrade by segment, format errors and hard-gate cases

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

A newer model often scores higher overall and still breaks something. One language gets worse, JSON comes back wrapped in a code fence, or a case that must never fail starts failing. An upgrade is judged on its parts, not its total.

The team's rules make the decision mechanical: block when a hard-gate case fails on the new model, when the new model returns more invalid JSON than the old one, or when any segment drops by more than a set amount. Otherwise ship.

Upgrades are also a chance to simplify. Prompts collect instructions written around an old model's weak spots. Running the new model with a shorter prompt shows whether those instructions still earn their place.

About the assignment

Score recorded outputs from the current and the new model by segment, count invalid JSON, list hard-gate failures, test the short prompt, and print the decision.

What you will do

  1. Score each segment separately for both models.
  2. Count json-segment replies that do not parse as a JSON object.
  3. List hard-gate cases that fail on the new model, then apply the rules in order.

Common mistakes

  • Deciding on the overall rate

    A better total can hide a much worse segment.

  • Counting a fenced reply as JSON

    A reply wrapped in a code fence does not parse, and downstream code will fail on it.

In a real test suite

Teams run this comparison on recorded outputs before switching traffic, then repeat it on a small share of live traffic. Writing the rules before seeing the numbers stops a good headline from talking everyone into a bad upgrade.

Related lessons

Next lessonLatency, throughput and token cost under loadAll lessons in the courseAI Testing for QA Engineers