Reporting, screenshots and logs

Reporting is how a test run tells people what happened: which tests passed, which failed, and why. When a test fails at night in CI, the report is all anyone has. Interviewers ask which report you produce, how screenshots get into it, how logging works and how results reach the team. A good answer covers what TestNG gives you for free, what you add, and how people use it.

What you get for free

Every TestNG run writes an HTML report (index.html), a shorter emailable-report.html and JUnit-style XML results (a text format for test results that other tools can read). When a test fails or is skipped, it also writes testng-failed.xml. When Maven runs the tests, Surefire puts its results in target/surefire-reports. The XML files matter most: Jenkins (a popular CI server, that is, a machine that runs your tests automatically) and most other CI tools read them for pass and fail counts, trends and lists of failed tests. GitHub Actions needs a small reporting action to do the same.

Reporter.log("Opened checkout") adds a line to TestNG's own report for that test, which is a simple start.

Richer reports

Most teams add a report library that comes from outside TestNG:

• Extent Reports builds an HTML report through its Java methods, usually from a TestNG listener. The listener creates an entry when a test starts, logs steps, marks the result and attaches screenshots.
• Allure works through an adapter (a small add-on that connects it to TestNG or JUnit) that writes result files during the run. The Allure command-line tool turns them into an HTML report with steps, attachments, history and grouping by feature.

Both are separate libraries, not part of TestNG. Name the one you used and what it gave the team.

Screenshot on failure

Put the screenshot code in a listener, so no test has to remember it. A listener is a class that TestNG calls on events such as a test failing:

public class FailureListener implements ITestListener {
    @Override
    public void onTestFailure(ITestResult result) {
        WebDriver driver = DriverManager.getDriver();          // per-thread driver
        if (driver == null) return;
        byte[] png = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
        String name = result.getMethod().getMethodName() + "_" + System.currentTimeMillis() + ".png";
        try {
            Path file = Paths.get("target", "screenshots", name);
            Files.createDirectories(file.getParent());
            Files.write(file, png);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }
}

Register it with @Listeners(FailureListener.class) or in testng.xml. Three details make it work. The listener runs before @AfterMethod quits the browser, so the page is still there. It takes the driver from the per-thread manager (a small class that gives each thread its own driver; see Parallel and CI runs), so in a parallel run each failing test captures its own browser. The file name holds the test name and time, so files never overwrite each other.

The same bytes go into Extent or Allure as an attachment. Saving the page URL, the page source and the browser console log next to the screenshot makes triage (finding out why a test failed) faster. An @AfterMethod that checks result.getStatus() == ITestResult.FAILURE is another valid place for this code.

Logging

Use a logging library, not System.out.println. Common choices are SLF4J with Logback, or Log4j 2. These are logging libraries that write messages with a level and a time:

private static final Logger log = LoggerFactory.getLogger(LoginPage.class);
log.info("Logging in as {}", user);
log.debug("Waiting for dashboard header");

The levels, from most to least detail, are TRACE, DEBUG, INFO, WARN and ERROR (Log4j 2 adds FATAL above ERROR), and the configuration file sets the level per package. An appender decides where log lines go: the console, a file, or a rolling file that starts fresh by size or date. Put the thread name in the log format, so lines from parallel tests can be told apart. Log page actions at INFO, waits at DEBUG and errors with detail at ERROR, and never log passwords or tokens.

Getting results to people

• CI publishes the XML results and keeps the HTML report as a build artifact (a file the CI job saves, such as a report) or a published page, even when tests failed.
• A short message goes to the team channel or by e-mail with the counts and a link.
• Each failure is triaged (sorted by cause): a product bug (raise it with the evidence), a script problem (fix it) or an environment problem.

What a useful failure report contains

A clear test name, the assertion message with expected and actual values, the steps, a screenshot, the URL, the environment and browser, and the build number. If a developer can act on it without rerunning the test, the report has done its job.

How to answer in an interview

Answer in three layers: "TestNG and Surefire write XML that Jenkins reads for trends. A listener feeds Allure with steps and a screenshot on every failure. Jenkins publishes the report after each run and posts a summary with the link." Then add one detail that shows you used it, such as the screenshot being taken before the browser quits.

Common mistakes

• Taking screenshots in @AfterMethod after the browser has already quit.
• Using a static driver in the listener, so a parallel run saves the wrong browser's screenshot.
• Fixed screenshot file names that overwrite each other.
• Publishing reports only when the build passes, which is exactly when nobody needs them.

Recap

TestNG and Surefire give you HTML and XML reports for free (plus testng-failed.xml after failures), and CI reads the XML. Extent or Allure adds readable reports with steps and screenshots. A listener takes the screenshot before teardown, a logging library records what happened, and CI always publishes the results.