How To Generate TestNG Reports In Jenkins?

Why Use TestNG in the First Place?

TestNG is quite popular among developers and testers for test creation. It offers several practical features like grouping, dependence, prioritization, ease of using multiple annotations, etc. If you are not a fan of TestNG, allow me to introduce you to a few of the several benefits offered by it, especially from Selenium perspective-

• It supports annotations.
• It has a flexible test configuration, ensuring that the code is multithread safe.
• Support for data-driven testing (with @DataProvider).
• Support for parameters.
• No more need for TestSuite and accompanies a powerful execution model.
• Supports a variety of tools and plugins, which include Jenkins, Eclipse, Maven, etc.
• Embeds BeanShell for further flexibility.
• It is designed to cover tests (almost all): Unit tests, Functional tests, End-to-end tests, Integration tests, etc.
• It uses more Java and OO concepts and features.

Importance of TestNG Annotations

Primarily, developers use annotation repeatedly to define a piece of code that is incorporated into the software or business logic used in the programming language Java to control the flow of methods. In Test Next Generation (TestNG), annotations play a significant role. When working on TestNG, testers need to consider the operations and uses of each annotation.

Although Annotations differ from project to project depending on their requirements, the execution flow remains intact for every single project. Below are different Annotations present in TestNG and their primary usage:

• @BeforeSuite: The annotated method will run before all tests in this suite have run.
• @AfterSuite: The annotated method will run after all tests in this suite have run.
• @AfterTest: The annotated method will run after all the test methods belonging to the classes inside the tag have run.
• @BeforeTest: The annotated method will run before any test method belonging to the classes inside the tag has run.
• @AfterGroups: The annotated method is a list of groups that the configuration method will run after. This method will run shortly after invoking the last test method that belongs to any of these groups.
• @BeforeGroups: The annotated method is a list of groups that the configuration method will run before. This method will run shortly before invoking the first test method that belongs to any of these groups.
• @AfterClass: The annotated method will run after all the current class test methods have been run.
• @BeforeClass: The annotated method will run before the first test method in the current class is invoked.
• @AfterMethod: The annotated method will run after each test method has been run.
• @BeforeMethod: The annotated method will run before each test method has been run.
• @Test: This annotated method is a part of a test case.

There are several benefits of the available TestNG Annotations–

• By looking up annotations, it describes the methods that it is interested in. Method names are also not limited to any pattern or format.
• We can pass annotations with additional parameters.
• Annotations are strongly typed so that the compiler can flag any errors right away.
• Test classes are no longer needed for extensions.

Watch this video to learn about the TestNG Annotations and how they help provide better structure and readability to the code.

Writing a Basic Test in TestNG

Now that we have essential elements in place let us see the primary steps involved in creating a TestNG test.

• Write the business logic of the test.
• Insert TestNG annotations in the code.
• Include the test information (e.g., the methods names, class names, group names, etc.) in a testng.xml file.
• Run TestNG.

Note: TestNG requires JDK 8 or higher. You can refer to our complete TestNG Tutorial to start from the basics.

Below is a straightforward test is written in TestNG:

package example1;
 
import org.testng.annotations.*;
 
public class SimpleTest {
 
 @BeforeClass
 public void setUp() {
   // here would be the code that will be invoked when this test is instantiated
 }
 
 @Test(groups = { "fast" })
 public void aFastTest() {
   System.out.println("Fast test executed");
 }
 
 @Test(groups = { "slow" })
 public void aSlowTest() {
    System.out.println("Slow test executed");
 }
 
}

Using @BeforeClass and @Test, i.e., after the test class is built & before any test method is run, the setUp() method will be invoked. We will run the group quickly in this example, so aFastTest() is invoked while a SlowTest() is skipped.

Things to Note:

• No need to implement an interface or extend a class.
• Even though the above example uses the JUnit conventions, it is the annotations that tell TestNG what our methods can be called any name you like.
• A test method could belong to one or multiple groups.

We can either invoke our test using the command line, the ant task (as shown below), or the XML file: once you have compiled your test class into the build directory.

<project default="test">
 
 <path id="cp">
   <pathelement location="lib/testng-testng-5.13.1.jar"/>
   <pathelement location="build"/>
 </path>
 
 <taskdef name="testng" classpathref="cp"
          classname="org.testng.TestNGAntTask" />
 
 <target name="test">
   <testng classpathref="cp" groups="fast">
     <classfileset dir="build" includes="example1/*.class"/>
   </testng>
 </target>
 
</project>

Use ant to invoke it:

c:> antBuildfile: build.xml

Output

The test results can be browsed using:

start test-output\index.html (on Windows)

If you don’t like this process of getting the test results, there’s an easier way to do it. It is also possible to generate the TestNG reports in Jenkins. There is a plugin Jenkins offers to accomplish this task. The plugin is the TestNG Results plugin. This next section will find out how to generate TestNG report in Jenkins with this plugin.

You might also like- Generate PHPUnit Coverage Report In HTML and XML

2M+ Devs and QAs rely on TestMu AI

Deliver immersive digital experiences with Next-Generation Mobile Apps and Cross Browser Testing Cloud

Sign up with googleSign up with Email

With TestNG certification, you can challenge your skills in performing automated testing with TestNG and take your career to the next level.

Here’s a short glimpse of the TestNG certification from TestMu AI:

TestNG Results Plugin

TestNG Results plugin enables you to publish the results of TestNG created using org.testng.reporters.XMLReporter.TestNG The resulting XML file contains more details than the XML file of the JUnit framework. In graph and table reports, this plugin reveals such additional information. The plugin also makes it possible to import the XML TestNG reports in Jenkins from each build.

Now getting to the ‘how’ part of ‘how to generate TestNG report in Jenkins’- data is parsed using the output that org.testng.reporters.XMLReporter generates. The results are shown with a trend graph, and all the specifics of the failed tests are also presented. And that’s how TestNG reports in Jenkins look like!

Let’s find out more about a Jenkins Selenium report.

Features of TestNG Results Plugin

A Jenkins Selenium report generated using TestNG Results plugin offers the following features-

• The page layouts are intentionally designed so that they look very similar to Jenkins’s JUnit plugin. Also known for the fact JUnit inspires them.
• You will see a summary of passed/fail checks and passed/fail configuration methods on the project/build tab. To see the test failure, it also enlists the failed test cases with a direct connection.

• The test methods in the class result summary page are separated from the configuration method.
• In the Package result summary, we can see the order in which test methods were run during the regression run.

• The pass/failed pattern of previous job results will be seen in different graphs.

• Able to hide or expand tables that display package and class results.

Watch this video to learn how TestNG has become one of the most robust test automation frameworks and all you need to know to get started with TestNG in Selenium.

Utilities of TestNG Results Plugin

Using this plugin is very straightforward once you have configured your build to run TestNG-based test cases and figured out how to generate TestNG report in Jenkins. Simply trigger Publish TestNG Results in Post-Build Behavior.

This option enables the following properties to be configured:

• TestNG XML report pattern- This is a pattern for the file name that can be used to find the TestNG XML report files (for example, */target/testng-results.xml*). The path is either an Ant-style pattern (e.g., fileset) or a list of files and folders separated by the characters’;:,’. It is also vital that TestNG be configured to generate an XML Jenkins Selenium report using _org.testng.reporters. This is to ensure XMLReporter for this plugin functions properly.
• The Escape Test Description string- If checked, while displaying test method results, the plugin escapes the description string that is associated with the test method. Unchecking allows us to use HTML tags to format the description, which is enabled by default.
• The Escape exception messages- If checked, the plugin escapes the test method’s exception messages. Unchecking allows you to use HTML tags to format the exception message, e.g., embed links in the text. This is enabled by default.
• Show Failed Builds- If selected, the plugin includes results from the failed builds in the trend graph. This is disabled by default.
  • In case this is a maven build, it is always better to configure the build step with -Dmaven.test.failure.ignore=true option. This results in the build with test failures being marked as Unstable, thus distinguishing it from the build that failed because of issues not related to the tests.
  • Also, even when this option is selected, failed builds with no results and aborted builds will not be displayed in graphs.
• Marking the build as a failure on a failed configuration- This marks the build as failed if there are any configuration method failures.
• Configuring thresholds for marking the build as Unstable/Failed- A Threshold Mode can be configured by the user as a number of tests or a percentage of tests and then configured separately for these values. Even if there are several test failures or label the build as failed, even if there is a single test method failure, this allows for situations where we want the builds to be marked successful.

9 Of The Best Reporting Tools For Selenium

How to Generate TestNG Reports in Jenkins?

Now let us see how to generate TestNG Report in Jenkins. To be able to do this, you need to have all the required tools in your system.

Prerequisites-

• TestNG Project
• Jenkins – You can check Jenkins installation here.
• TestNG Results Plugin in Jenkins

Once you have all this installed, we can get started with a TestNG Project.

How To Create a TestNG Project?

In order to generate TestNG reports in Jenkins, we need to start from scratch. First, we will create a TestNG project. To create a TestNG project, you will need to follow the steps mentioned below-

Step 1: Open Eclipse and create a Java class.

Step 2: Keep all the library files in a folder.

Step 3: Write a Java program.

The below snippet highlights all the three steps mentioned above.

Step 4: Now convert the Java Program into TestNG.

Alternative Method- Another way you can create a TestNG Project is by creating a new TestNG project all from the beginning.

Step 1: Click File –> New –> Java Project

Step 2: Give it an arbitrary Project Name, say “FirstTestNGProject” and then click Next.

Step 3: Next step is to start importing the TestNG Libraries into the project. Click on the “Libraries” tab –> “Add Library”

Step 4: Choose “TestNG” on the Add Library dialog, and click Next.

Step 5: Click on Finish.

It should be noticed that TestNG is included on the Libraries list.

Step 6: The JAR files containing the Selenium API will now be added. These files are located in the Java client driver we downloaded from http://docs.seleniumhq.org/download/.

Then, navigate to where the Selenium JAR files are placed.

After adding the external JARs, the screen should appear like this.

Step 7: Click on Finish and confirm that our FirstTestNGProject is visible on Eclipse’s Package Explorer window.

After we have the TestNG project, the next step is to install Jenkins. You can refer to the guided steps for Jenkins installation in our Jenkins Tutorial.

Now that we have TestNG Project and Jenkins platform ready, the next step is to install TestNG Reports Plugin in Jenkins.

Installing TestNG Reports Plugin In Jenkins

The next and the most crucial step to know how to generate TestNG report in Jenkins is to install the TestNG Reports plugin. To install the plugin in Jenkins, follow these steps-

• Open Jenkins (localhost:8080).
• Click on ‘Manage Jenkins‘ → Click on ‘Manage Plugins‘

• Click on ‘Available‘ tab
• Select ‘TestNG Results Plugin‘
• Click on ‘Install Without Restart‘

• Open the ‘TestNGProject’ Job created earlier.
• Click on ‘Configure’ and scroll down to select ‘Post Build Actions’
• Click on ‘Add Post Build Actions’ dropdown list.

• And select ‘Publish TestNG Results‘.

• In the TestNG XML Report Pattern edit box enter ‘**/testng-results.xml‘ and click ‘Save‘

• We have already created a new project ‘TestNGProject ‘to run TestNG Tests and generate TestNG Reports after execution using Jenkins.
• Click on the ‘Build Now‘ button to execute it. This will invoke testng.xml from the batch file.
• Right-click on Build Number and click on Console Output to see the result.
• Once the execution is completed, there will be a link to view ‘TestNG Results’

There you go- that’s how to generate TestNG Report in Jenkins!

Alternative Method To Generate TestNG Reports in Jenkins

There’s another way to set up the TestNG reports in Jenkins Dashboard.

• Login into Jenkins.
• Manage Jenkins and Install TestNG Result Plugin.
• Please make sure that you restart Jenkins after the plugin installation.

• Next, go to Jenkins Home Page → Create New Jenkins Job and in Post-Build Action select → Publish TestNg Result.

• Now either give the complete path of your testng-result.xml or type **/testng-results.xml in TestNG XML report pattern to search inside your project workspace.

• Execute your Jenkins Job, and Now your Jenkins Selenium report or ‘TestNG Results’ would be visible in Your Jenkins Dashboard.