Skip to content
CodeKruPro

API tests around the model

13/99

Module · A deterministic harness

beginner

about 40 min

What this lesson teaches

The idea

Most incidents in AI features happen in the API around the model: no key check, an oversized prompt cut silently, an error page instead of JSON, a stream that stops early.

Why a tester cares

These are plain HTTP tests: fast, cheap and stable. Each clause of the API contract becomes one test.

Words you will meet

API contract:
the written promise of how an API behaves for each kind of request.
HTTP status:
the number on every response, such as 200 for success, 401 for a missing key or 429 for too many requests.
Retry-After:
a response header telling the caller how long to wait before trying again.

What you will do

  1. 1Read the contract
  2. 2Run the starter
  3. 3Write a test per clause
  4. 4Submit
Jump to practice

Most production incidents in AI features are not the model being wrong. They are the API around it: a missing key that still gets an answer, a 500 that returns an HTML page your code cannot parse, a stream that stops halfway and is shown as complete. These are ordinary API tests, and they are the cheapest tests you will write in this course.

RefundBot's chat endpoint has a written contract. Your tests check it on several builds of the endpoint; some builds break one clause.

The contract

  • A request without a valid key gets 401 with a JSON error whose type is invalid_request_error.
  • A prompt longer than 20,000 characters gets 400 with error code context_length_exceeded. It is never cut down and answered.
  • When overloaded it returns 429 with a Retry-After header in seconds.
  • Server errors return 500 with a JSON body whose error type is server_error.
  • A normal reply is never empty, and a streamed reply ends with data: [DONE].

Making faults happen on purpose

You cannot wait for a real outage. The practice endpoint has a test hook: send the header X-Practice-Fault with 429, 500 or slow (a 4-second delay), and it produces that fault. Real teams build the same kind of hook, or use a mock server, so failure paths can be tested on demand.

Your task

Write one test per clause of the contract above, plus a test that your client gives up on a slow reply instead of hanging. You can work in Python with requests, or in Java with JUnit 5 and REST Assured.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

Write it in
Both are graded the same way.
against the practice bot

How this is checked: When you submit, your tests run against 8 versions of RefundBot: 2 correct and 6 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

API contract testing for an LLM chat endpoint: keys, limits, rate limits and streams

AI Testing for QA EngineersBeginnerA deterministic harness

How it works

Most incidents in AI features happen in the API around the model: an endpoint that answers without a key, an oversized prompt silently truncated instead of refused, a 429 without a Retry-After header, a 500 that returns an HTML page, an empty reply with a success status, or a stream that stops without its final event.

Each clause of a written contract becomes one test. These are plain HTTP tests, which makes them fast, cheap and stable, and they run in Python with requests or in Java with REST Assured.

About the assignment

Write one test per clause of RefundBot's contract, so each faulty build fails exactly where it breaks a clause and correct builds pass.

What you will do

  1. Read the contract and list its clauses.
  2. Run the starter to see the request helpers in action.
  3. Use the fault header from the lesson to trigger rate limits and server errors on demand.

Common mistakes

  • Testing the happy path only

    Most clauses describe failures. Trigger each failure deliberately.

  • Asserting only the status code

    Several clauses are about the body or headers: the error type, a Retry-After value, the final stream event.

In a real test suite

Contract tests are the first suite to automate for a new AI feature because they need no judgement about answer quality, and they catch the outages customers notice first.

Keep each test focused on one clause and name it after the clause. When a build fails, the test name alone should tell an engineer which promise in the contract was broken, without opening the log.

Related lessons

Next lessonMetamorphic tests when no oracle existsAll lessons in the courseAI Testing for QA Engineers