TestNG Reports: Default HTML, Emailable, and Jenkins

TestNG Results Plugin

TestNG Results plugin enables you to publish the results of TestNG created using org.testng.reporters.XMLReporter.TestNG The resulting XML file contains more details than the XML file of the JUnit framework. In graph and table reports, this plugin reveals such additional information. The plugin also makes it possible to import the XML TestNG reports in Jenkins from each build.

Now getting to the ‘how’ part of ‘how to generate TestNG report in Jenkins’- data is parsed using the output that org.testng.reporters.XMLReporter generates. The results are shown with a trend graph, and all the specifics of the failed tests are also presented. And that’s how TestNG reports in Jenkins look like!

Let’s find out more about a Jenkins Selenium report.

Features of TestNG Results Plugin

A Jenkins Selenium report generated using TestNG Results plugin offers the following features-

• The page layouts are intentionally designed so that they look very similar to Jenkins’s JUnit plugin. Also known for the fact JUnit inspires them.
• You will see a summary of passed/fail checks and passed/fail configuration methods on the project/build tab. To see the test failure, it also enlists the failed test cases with a direct connection.

• The test methods in the class result summary page are separated from the configuration method.
• In the Package result summary, we can see the order in which test methods were run during the regression run.

• The pass/failed pattern of previous job results will be seen in different graphs.

• Able to hide or expand tables that display package and class results.

Watch this video to learn how TestNG has become one of the most robust test automation frameworks and all you need to know to get started with TestNG in Selenium.

Utilities of TestNG Results Plugin

Using this plugin is very straightforward once you have configured your build to run TestNG-based test cases and figured out how to generate TestNG report in Jenkins. Simply trigger Publish TestNG Results in Post-Build Behavior.

This option enables the following properties to be configured:

• TestNG XML report pattern- This is a pattern for the file name that can be used to find the TestNG XML report files (for example, */target/testng-results.xml*). The path is either an Ant-style pattern (e.g., fileset) or a list of files and folders separated by the characters’;:,’. It is also vital that TestNG be configured to generate an XML Jenkins Selenium report using _org.testng.reporters. This is to ensure XMLReporter for this plugin functions properly.
• The Escape Test Description string- If checked, while displaying test method results, the plugin escapes the description string that is associated with the test method. Unchecking allows us to use HTML tags to format the description, which is enabled by default.
• The Escape exception messages- If checked, the plugin escapes the test method’s exception messages. Unchecking allows you to use HTML tags to format the exception message, e.g., embed links in the text. This is enabled by default.
• Show Failed Builds- If selected, the plugin includes results from the failed builds in the trend graph. This is disabled by default.
  • In case this is a maven build, it is always better to configure the build step with -Dmaven.test.failure.ignore=true option. This results in the build with test failures being marked as Unstable, thus distinguishing it from the build that failed because of issues not related to the tests.
  • Also, even when this option is selected, failed builds with no results and aborted builds will not be displayed in graphs.
• Marking the build as a failure on a failed configuration- This marks the build as failed if there are any configuration method failures.
• Configuring thresholds for marking the build as Unstable/Failed- A Threshold Mode can be configured by the user as a number of tests or a percentage of tests and then configured separately for these values. Even if there are several test failures or label the build as failed, even if there is a single test method failure, this allows for situations where we want the builds to be marked successful.

9 Of The Best Reporting Tools For Selenium

TestNG Default Reports: index.html and emailable-report.html

Before Jenkins enters the picture, TestNG already writes three reports after every run, into test-output when you run from the IDE or the TestNG command line, and into target/surefire-reports when Maven Surefire runs the suite:

• index.html is the full report: one page per suite with the pass, fail, and skip counts, every test method with its duration and stack trace on failure, the Reporter.log output, and the testng.xml that ran.
• emailable-report.html is the same data in a single self-contained file with inline styles, made to be attached to an email or posted in a chat channel.
• testng-results.xml is the machine-readable version. CI servers, dashboards, and the Jenkins plugin below read this file, not the HTML.

These are enough for a developer reading results on their own machine. Jenkins adds what they lack: history across builds, trend graphs, and a link on every build page, which is what the rest of this chapter sets up.

Custom TestNG Reports With ExtentReports and Allure

The default reports answer what passed, not what happened. When a failure needs a screenshot, a step-by-step log, or a trend across runs, teams plug a reporting library into the TestNG listener API. Two libraries cover almost every team.

ExtentReports builds a single HTML dashboard with per-test logs, categories, and attached screenshots. Add the dependency, then write a listener that opens a test entry when a method starts and records the outcome when it ends. The TestNG listeners chapter explains the ITestListener interface the class below implements:

<dependency>
    <groupId>com.aventstack</groupId>
    <artifactId>extentreports</artifactId>
    <version>5.1.2</version>
</dependency>

import com.aventstack.extentreports.ExtentReports;
import com.aventstack.extentreports.ExtentTest;
import com.aventstack.extentreports.reporter.ExtentSparkReporter;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class ExtentListener implements ITestListener {
    private static final ExtentReports extent = new ExtentReports();
    private static final ThreadLocal<ExtentTest> test = new ThreadLocal<>();

    @Override
    public void onStart(ITestContext context) {
        extent.attachReporter(new ExtentSparkReporter("test-output/extent-report.html"));
    }

    @Override
    public void onTestStart(ITestResult result) {
        test.set(extent.createTest(result.getMethod().getMethodName()));
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        test.get().pass("Passed");
    }

    @Override
    public void onTestFailure(ITestResult result) {
        test.get().fail(result.getThrowable());
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        test.get().skip(result.getThrowable());
    }

    @Override
    public void onFinish(ITestContext context) {
        extent.flush();
    }
}

Register the listener once, either with @Listeners(ExtentListener.class) on a test class or with a listeners block in testng.xml, and every run writes test-output/extent-report.html next to the default reports.

Allure takes the other approach: a small adapter records results as JSON while the suite runs, and a separate command turns them into an interactive site with history, retries, and per-step timing. Add the allure-testng dependency, run the suite, then generate the report:

# pom.xml: io.qameta.allure:allure-testng:3.0.0 (test scope)
mvn test
allure serve target/allure-results

Both libraries have Jenkins plugins that archive and display the report per build, so the Jenkins steps in the rest of this chapter apply to them as well as to the TestNG Results plugin.

How to Generate TestNG Reports in Jenkins?

Now let us see how to generate TestNG Report in Jenkins. To be able to do this, you need to have all the required tools in your system.

Prerequisites-

• TestNG Project
• Jenkins – You can check Jenkins installation here.
• TestNG Results Plugin in Jenkins

Once you have all this installed, we can get started with a TestNG Project.

Installing TestNG Reports Plugin In Jenkins

The next and the most crucial step to know how to generate TestNG report in Jenkins is to install the TestNG Reports plugin. To install the plugin in Jenkins, follow these steps-

• Open Jenkins (localhost:8080).
• Click on ‘Manage Jenkins‘ → Click on ‘Manage Plugins‘

• Click on ‘Available‘ tab
• Select ‘TestNG Results Plugin‘
• Click on ‘Install Without Restart‘

• Open the ‘TestNGProject’ Job created earlier.
• Click on ‘Configure’ and scroll down to select ‘Post Build Actions’
• Click on ‘Add Post Build Actions’ dropdown list.

• And select ‘Publish TestNG Results‘.

• In the TestNG XML Report Pattern edit box enter ‘**/testng-results.xml‘ and click ‘Save‘

• We have already created a new project ‘TestNGProject ‘to run TestNG Tests and generate TestNG Reports after execution using Jenkins.
• Click on the ‘Build Now‘ button to execute it. This will invoke testng.xml from the batch file.
• Right-click on Build Number and click on Console Output to see the result.
• Once the execution is completed, there will be a link to view ‘TestNG Results’

There you go- that’s how to generate TestNG Report in Jenkins!

Alternative Method To Generate TestNG Reports in Jenkins

There’s another way to set up the TestNG reports in Jenkins Dashboard.

• Login into Jenkins.
• Manage Jenkins and Install TestNG Result Plugin.
• Please make sure that you restart Jenkins after the plugin installation.

• Next, go to Jenkins Home Page → Create New Jenkins Job and in Post-Build Action select → Publish TestNg Result.

• Now either give the complete path of your testng-result.xml or type **/testng-results.xml in TestNG XML report pattern to search inside your project workspace.

• Execute your Jenkins Job, and Now your Jenkins Selenium report or ‘TestNG Results’ would be visible in Your Jenkins Dashboard.