Skip to content
CodeKruPro

Latency, throughput and token cost under load

83/99

Module · Release, production and operations

advanced

about 40 min

What this lesson teaches

The idea

An AI feature that answers well for one tester can fall over for a hundred customers. A load test sends many requests at once and measures speed, errors and cost.

Why a tester cares

Rate limits shape the result, and replies cut off by a length limit look like successes. Only careful retries and the finish reason give honest numbers.

Words you will meet

Latency:
how long a reply takes.
95th percentile:
the time 95% of replies beat; it shows the slow tail that an average hides.
Rate limit:
a cap on requests at once; above it the server answers 429.
Jitter:
a small random wait added before a retry so clients do not collide again.

What you will do

  1. 1Send 50 at once
  2. 2Retry 429s properly
  3. 3Measure
  4. 4Submit
Jump to practice

An AI feature that answers well for one tester can fall over for a hundred customers. A load test sends many requests at once and reports what a product owner needs:

  • p50 and p95 latency of the requests that succeeded, in milliseconds. Use the nearest-rank method: sort the values, take the one at position ceil(p/100 × n).
  • Error rate: the share of all HTTP responses that were not 200, retries included.
  • Cut-off replies: answers with finish_reason == "length". They return 200 and look like successes.
  • Cost per 1,000 requests from the token usage, at Rs 0.05 per 1,000 prompt tokens and Rs 0.20 per 1,000 completion tokens.

This endpoint allows 8 requests in flight. Above that it answers 429 with a Retry-After header. A load test that gives up on a 429 measures nothing useful. One that retries without waiting makes things worse, and so do 50 clients that all retry at the same instant: add random jitter to each wait.

Your task

Send all 50 questions at once, retry every 429 after Retry-After, and print the figures. They are checked against the practice server's own log of your run. On submit it also runs against a busier endpoint.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

against the endpoint

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Load test an LLM API: p50 and p95 latency, 429 retries, cut-off replies and cost per 1,000 requests

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

An AI feature that answers well for one tester can fall over for a hundred customers. A load test sends many requests at once and reports what a product owner needs: median and 95th percentile latency of successful requests, the share of all responses that were errors, replies cut off by a length limit, and the cost per thousand requests from token usage.

Rate limits shape the result. This endpoint allows a fixed number of requests in flight and answers 429 with a Retry-After header above that. A test that gives up on a 429 measures nothing useful, one that retries without waiting makes things worse, and fifty clients retrying at the same instant collide again, which is why waits get random jitter.

Cut-off replies are the quiet failure. They return 200 and look like successes, so only the finish reason reveals them.

About the assignment

Send all the questions at once, retry 429s properly, and print the figures, which are checked against the practice server's own log of your run.

What you will do

  1. Disable the SDK's own retries, so every 429 your script receives is counted.
  2. Send the requests in parallel and retry each 429 after its delay plus jitter.
  3. Compute latency percentiles from successful requests, the error rate from all responses, and the cost from usage.

Common mistakes

  • Counting only final failures

    The error rate counts every non-200 response, retries included.

  • Timing the whole run

    Latency is per request. Time each successful call.

In a real test suite

Load results feed capacity planning and pricing. Teams rerun the test after model or prompt changes, because a longer prompt raises both latency and cost per request.

Related lessons

Next lessonRead a LangSmith exportAll lessons in the courseAI Testing for QA Engineers