Build tools and running tests from the command line

A CI server (a computer that runs builds and tests automatically; see the Jenkins topic) runs your Python tests on a clean machine with a few commands, reads pytest's exit code (a number the command returns when it ends; 0 means success) and collects a report file. Interviewers check that you know those commands, because "it works in my editor" is not enough: how is the environment built, how do you choose which tests run, and how does a failure reach the CI server? This article covers each.

A clean environment: venv and requirements

A virtual environment is a private folder of Python packages for one project, separate from the system and from other projects:

python3 -m venv .venv
source .venv/bin/activate          # Linux and macOS
.venv\Scripts\activate             # Windows
pip install -r requirements.txt

requirements.txt lists the packages with pinned versions (an exact version number for each package), for example pytest==8.3.5, requests==2.32.3, selenium==4.47.0 (the numbers are examples; pin whatever your team has tested). Without pins, a fresh CI agent (a machine that runs the build) installs the newest release of everything, and the nightly run can break although nobody changed the code. pip freeze > requirements.txt records the exact versions of a known-good environment. Commit requirements.txt and add .venv/ to .gitignore.

Anything your command line relies on must be in that file. A classic CI failure: pytest -n 4 works on a laptop but fails on the agent with "unrecognized arguments: -n", because the pytest-xdist plugin (an add-on that runs tests in parallel) was installed by hand and never added to the requirements.

Configuration: pytest.ini or pyproject.toml

[pytest]
testpaths = tests
addopts = --strict-markers
markers =
    smoke: quick checks run on every pull request
    slow: long tests run nightly

A marker is a label on a test, such as @pytest.mark.smoke. Registering markers matters: an unregistered marker gives a PytestUnknownMarkWarning, and with --strict-markers it stops collection, so a typo like smokee cannot silently drop tests. (Collection is the step where pytest finds the tests to run.) Shared fixtures (setup code that tests can ask for) go in conftest.py.

Choosing which tests run

pytest                                         # everything under testpaths
pytest tests/test_login.py                     # one file
pytest tests/test_login.py::test_invalid_password    # one test (node id)
pytest "tests/test_login.py::test_param[b]"    # one parametrized case
pytest -k invalid                              # names containing "invalid"
pytest -k "invalid and not locked"             # name expressions
pytest -m smoke                                # tests marked smoke
pytest -m "smoke and not slow"                 # marker expressions

The text after :: is part of the node id, the full name pytest gives one test. -k matches names; -m matches markers.

Controlling the run

• -x stops at the first failure, and --maxfail=3 after three.
• --lf (last failed) reruns only the tests that failed last time, using the cache in .pytest_cache. Great locally; in CI each run usually starts without that cache.
• -v lists each test, -q is quieter, and -s shows print output live.
• -n 4 or -n auto runs tests at the same time in several worker processes (separate copies of Python), with the pytest-xdist plugin.
• --junitxml=reports/junit.xml writes a JUnit XML report that Jenkins, GitHub Actions and test management tools understand.

Prefer python -m pytest when tests import project packages from the repository root: it adds the current directory to the import path (the list of folders Python searches when you write import), which plain pytest does not. Otherwise set the path in the pytest settings, or install the project as a package.

Exit codes: how CI knows

0   all collected tests passed
1   some tests failed
2   interrupted (also errors during collection)
3   internal pytest error
4   command-line usage error (unknown option, broken conftest import)
5   no tests were collected

Any code other than 0 makes the CI step fail. Code 5 deserves attention: a misspelled marker or path means nothing ran, and you want that to be red. Code 4 with "ImportError while loading conftest" usually means a missing package on the agent.

Passing settings into tests

Environment variables are the simplest way: BASE_URL=https://staging.example.com BROWSER=firefox pytest -m smoke, read in a fixture with os.environ.get("BROWSER", "chrome"). Secrets always arrive this way, from the CI secret store. Custom options such as --browser can be added with the pytest_addoption hook (a special function that pytest calls at start-up) in conftest.py.

What CI runs

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python -m pytest -m regression -n 4 --junitxml=reports/junit.xml

A pipeline is the list of steps the CI server runs. It then publishes reports/junit.xml, even when tests fail.

How to answer in an interview

Walk through the four CI commands above and say why each is there: a clean environment, pinned packages, the right selection, and a report plus an exit code for the server. Then mention one failure you know how to read, such as exit code 5 meaning no tests were collected.

Common mistakes

• Unpinned requirements, so the nightly run changes by itself.
• Mixing up -k and -m.
• Plugins installed by hand on a laptop and missing in CI.
• Publishing the report only when tests pass.

Recap

Build a virtual environment, install pinned requirements and configure pytest once. Select tests by path, node id, -k or -m; control the run with -x, --lf and -n; and give CI a JUnit report. The exit code decides the step: 0 passes, anything else fails.