Reporting, screenshots and logs

Reporting is how a test run tells people what happened: which tests passed, which failed, and why. When a test fails in CI, nobody watches the browser, so the report, the screenshot and the log are the only evidence. Interviewers ask which report your framework produces, how a screenshot gets attached to a failed test, and how logging works. pytest gives you the building blocks, and plugins plus one well-known hook pattern do the rest.

Built-in output

pytest --junitxml=results.xml writes JUnit-style XML (a text format for test results that other tools can read), the format CI servers read for counts, trends and failed-test lists. A CI server is a machine that runs your tests automatically, for example after every code change. -ra adds a short summary of skips, failures and errors at the end of the console output. pytest writes no HTML report by itself.

HTML reports

• pytest-html, a plugin you install separately: pytest --html=report.html --self-contained-html gives one file with the results, logs and any images you attach.
• allure-pytest, also a separate plugin: pytest --alluredir allure-results writes result files, and the Allure command-line tool builds the report with allure generate or opens it with allure serve. Allure adds steps, attachments, history and grouping by feature.

Screenshot on failure

A fixture cannot see the test's result by itself. The pattern from the pytest documentation uses a hook in conftest.py. A hook is a function with a fixed name that pytest calls at a fixed point in the run. This one stores each phase's report on the test item (pytest's object for one test), and the driver fixture reads it during teardown:

import pytest

@pytest.hookimpl(wrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
    rep = yield
    setattr(item, "rep_" + rep.when, rep)     # rep_setup, rep_call, rep_teardown
    return rep

@pytest.fixture
def driver(request):
    d = create_driver(request.config)
    yield d
    rep = getattr(request.node, "rep_call", None)
    if rep is not None and rep.failed:
        png = d.get_screenshot_as_png()
        attach_to_report(request.node.name, png)    # your helper: file, Allure or pytest-html
    d.quit()

In the hook, yield lets pytest build the report first and hands it back to your code, and the next line saves it on the item. pytest runs every test in three phases, and rep.when names them: "setup", "call" and "teardown". The call report covers the test body. A failed setup means a fixture failed, which is a different problem from a failed assertion. The screenshot is taken before quit(), while the page is still open. Older code writes hookwrapper=True with outcome = yield and outcome.get_result(); both styles work in current pytest. With Allure the helper is one call: allure.attach(png, name="screenshot", attachment_type=allure.attachment_type.PNG).

Logging

Use the standard logging module, not print:

import logging
log = logging.getLogger(__name__)

def login_as(self, user, password):
    log.info("Logging in as %s", user)
    ...

The levels are DEBUG, INFO, WARNING, ERROR and CRITICAL. pytest captures log records and shows them in the report of each failing test. Settings in the config file (pytest.ini) control more:

[pytest]
log_cli = true
log_cli_level = INFO
log_file = logs/run.log
log_file_level = DEBUG

log_cli shows records live in the console, and log_file keeps a full file. The built-in caplog fixture lets a test check what was logged. Log page actions at INFO and waits at DEBUG, and never log passwords or tokens.

Getting results to people

• CI publishes the JUnit XML and keeps the HTML or Allure report as an artifact (a file the CI job saves) even when tests failed.
• A short summary with counts and a link goes to the team channel.
• Failures are sorted into product bugs, script problems and environment problems, and the verdict is written down.

With pytest-xdist (the plugin that runs tests in parallel), results from all workers end up in one JUnit file and one report. File names you create yourself, such as screenshots and downloads, must include the test name or worker id to stay unique.

What a useful failure report contains

A clear test name with its parametrize id, the assertion with both values, the steps or logs, a screenshot, the URL, and the environment, browser and build. If a developer can act on it without rerunning the test, the report works.

How to answer in an interview

Answer in three layers: "pytest writes JUnit XML for CI. allure-pytest gives a readable report, and a pytest_runtest_makereport hook plus the driver fixture attach a screenshot to every failure. CI publishes the report after each run and posts the link." If they ask how the fixture knows the test failed, explain that the hook stores the call report on the test item.

Common mistakes

• Trying to read the test result inside a fixture without the hook.
• Taking the screenshot after quit().
• Checking the setup report instead of the call report.
• Calling pytest-html "built in". It is a plugin you install.

Recap

pytest writes JUnit XML for CI, and plugins such as pytest-html and allure-pytest make reports for people. A pytest_runtest_makereport hook stores each phase's result, so the driver fixture can take a screenshot before quitting. The logging module records what happened, and CI always publishes the results.