Skip to content
CodeKruPro

MCP contract test: pin and diff tools

67/99

Module · Testing agents, chat UIs and extractors

advanced

about 35 min

What this lesson teaches

The idea

An agent trusts the tools an MCP server lists, including their descriptions, which the model reads as instructions. A server can quietly change them after approval.

Why a tester cares

A contract test pins the approved tool list in a file and fails the build when the live list changes.

Words you will meet

MCP server:
a service that offers tools to an AI agent.
Rug pull:
a tool changed after it was approved, to steer the agent.
Lock file:
a saved copy of what was approved, to compare against.
Contract test:
a test that the interface still matches what was agreed.

What you will do

  1. 1Read the pinned tools
  2. 2Diff names
  3. 3Diff descriptions and schemas
  4. 4Submit
Jump to practice

An agent trusts the tools an MCP (Model Context Protocol) server lists: their names, their descriptions, and their input schemas. The model reads the descriptions as guidance. So a server that quietly changes a description after your team approved it can steer the agent, a pattern known as a rug pull, and a new tool can appear that nobody reviewed.

The defence is an ordinary contract test. The team pins the approved tools/list in tools.lock.json, and CI fails when the live list differs. Version numbers may change; the tools may not, without a new approval.

Your task

Write the contract test against ToolShelf, a mock MCP server that speaks JSON-RPC at /mcp. On submit it runs against the approved server, a harmless version bump, a changed description, a new tool and a changed schema.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

Write it in
Both are graded the same way.
against ToolShelf

How this is checked: When you submit, your tests run against 5 versions of ToolShelf: 2 correct and 3 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

MCP server contract test for tool descriptions and schemas: catching a rug pull

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

An agent trusts the tools an MCP server lists: their names, descriptions and input schemas. The model reads descriptions as guidance, so a server that quietly changes a description after approval can steer the agent. That is a rug pull, and a new, unreviewed tool can appear the same way.

The defence is an ordinary contract test. The team pins the approved tool list in a lock file, and CI fails when the live list differs in names, descriptions or schemas. Version numbers may change; the tools may not, without a new approval.

About the assignment

Write a contract test for ToolShelf that compares its live tool list with the pinned file and fails on any change to tools, descriptions or schemas, but not on a version bump.

What you will do

  1. Call the server's tool list over JSON-RPC.
  2. Compare tool names, then each description and input schema, with the lock file.
  3. Ignore the server version, and report exactly what differs.

Common mistakes

  • Failing on the version number

    A harmless version bump is not a contract change.

  • Comparing names only

    A rug pull keeps the name and changes the description.

In a real test suite

Pinning third-party tool definitions is becoming standard practice for teams that let agents use external MCP servers, in the same way teams pin package versions and review lock file changes.

Store the lock file in the repository and require a reviewer's approval to change it. The approval is the moment a person reads the new description and decides whether the agent should trust it.

Related lessons

Next lessonTest the chat windowAll lessons in the courseAI Testing for QA Engineers