Skip to content
CodeKruPro

Python for testers: JSON and safe parsing

8/99

Module · Python and LLM APIs bridge

beginner

about 20 min

What this lesson teaches

The idea

Every chatbot API returns JSON. A test harness reads three things from it: the reply text, how the reply finished, and how many tokens it used.

Why a tester cares

Real responses are not always complete. Code that assumes every field is there crashes on the first odd reply, and a crashed harness reports nothing.

Words you will meet

JSON:
a text format for data, made of named fields such as "text": "Hello".
Token:
a small piece of text the model counts and charges for, roughly three quarters of a word.
Defensive parsing:
checking that a field exists and has the right type before using it.

What you will do

  1. 1Run the starter
  2. 2Handle missing fields
  3. 3Submit
Jump to practice

Every chat model API answers with JSON. A test harness reads the reply text, how it finished, and how many tokens it used. Real responses are not always complete: an error response has no choices, a tool call has no text, and some gateways drop the usage block. A harness that crashes on those is a harness nobody trusts.

Your task

Finish parse_reply in reply_tools.py. It receives one response as a Python dict and returns a dict with three keys:

  • content: the first choice's message text;
  • finish_reason: the first choice's finish reason;
  • total_tokens: from the usage block.

Any value that is missing, or is not the expected type (text for the first two, a whole number for the last), becomes None. The function must never raise an exception.

Run uses the three sample responses shown below. Submit also tries responses you have not seen, with parts missing.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
responses.json (what Run uses)Show
[
  {
    "id": "chatcmpl-1",
    "object": "chat.completion",
    "model": "acme-practice-1",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "You have 30 days from delivery to request a return."
        },
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 10,
      "completion_tokens": 28,
      "total_tokens": 38
    }
  },
  {
    "id": "chatcmpl-1",
    "object": "chat.completion",
    "model": "acme-practice-1",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "Gift cards cannot be refunded."
        },
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 10,
      "completion_tokens": 14,
      "total_tokens": 24
    }
  },
  {
    "id": "chatcmpl-1",
    "object": "chat.completion",
    "model": "acme-practice-1",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "Refunds go back to your original payment method within 5 to"
        },
        "finish_reason": "length"
      }
    ],
    "usage": {
      "prompt_tokens": 10,
      "completion_tokens": 40,
      "total_tokens": 50
    }
  }
]
1

Your code

against the practice bot

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Parse an LLM API JSON response safely in Python without crashing on missing fields

AI Testing for QA EngineersBeginnerPython and LLM APIs bridge

How it works

Every chat completion API returns JSON, and a harness reads three things from it: the reply text, how the generation finished, and how many tokens it used. Real responses are not always complete. Error bodies have no choices, tool calls have no text, and some gateways strip the usage block.

Defensive parsing means checking each step before using it: that a value exists, that it has the expected type, and that a list is not empty before taking its first item. Code that assumes the happy path crashes on the first odd response, and a crashed harness reports nothing at all.

About the assignment

Finish parse_reply so that it returns the reply text, the finish reason and the total token count from one response, using None for anything missing or of the wrong type, and never raising.

What you will do

  1. Run the starter on the three sample responses and note where it fails.
  2. Check each level of nesting with isinstance before reading inside it.
  3. Treat a value of the wrong type the same as a missing one.

Common mistakes

  • Catching every exception

    A blanket try and except hides bugs in your own code. Check types explicitly so the function returns what it should.

  • Accepting a boolean as a number

    In Python, True is an instance of int. A token count should be a real whole number, so rule booleans out.

In a real test suite

Robust parsing is what lets a harness keep counting when one call in a thousand returns something strange. Teams log those odd responses separately instead of letting them crash the run or quietly count as passes.

Related lessons

Next lessonCall an LLM API with the OpenAI SDKAll lessons in the courseAI Testing for QA Engineers