How To Generate Code Coverage Report Using JaCoCo-Maven Plugin

What is Code Coverage?

Code Coverage is a metric used in software development to represent the extent to which an application’s source code is executed while a test suite is run. To see and evaluate the code coverage of a software application, a report is produced. Then, code quality may be ensured using this code coverage report.

The JaCoCo-Maven plugin is used to generate code coverage reports. Source code with high code coverage has more of its code executed during testing. For example, if the software you are testing contains 100 lines of code and the number of code lines validated in the software is 90, then the code coverage percentage of that software application will be 90%.

Benefits of Code Coverage

Code coverage is a very useful metric for developers, testers, and QA engineers. Here are some of the salient benefits of code coverage:

• Code coverage reports provide useful insights in detection and elimination of dead code. This can be avoided by following implementation best-practices which in turn results in improved code maintainability and better product quality.
• Quality assurance can help detect code that has not been covered using the test implementation.
• Developers can finish the software development process faster, increasing their productivity, scalability, and efficiency. This results in reduced Time to Market (TTM).

As we know, code coverage is very important for every software product. However, if you are a developer or a tester, you can cure your interest in code coverage via the article: Code Coverage vs Test Coverage, which one is better?

Now that we have done a quick recap about the integral aspects of code coverage, let’s deep dive into our core topic i.e. generating code coverage reports using Jacoco Maven plugin.

What is JaCoCo-Maven Plugin?

JaCoCo-Maven (abbreviation for Java Code Coverage) plugin is an open-source code coverage tool for Java. It creates code coverage reports and integrates well with IDEs(Integrated development environments) like Eclipse IDE.

It also integrates smoothly with CI/CD tools (e.g. Jenkins, Circle CI, etc.) and project management tools (e.g. SonarQube, etc.). It is a part of the Eclipse Foundation and has replaced the EclEmma code coverage tool in Eclipse.

How does JaCoCo-Maven Plugin work?

• The JaCoCo-Maven plugin runs the coverage by instrumenting Java code through a runtime agent. In simple terms, you attach this agent to a JVM (Java Virtual Machine) when it starts. This agent is termed as JaCoCo agent. The first execution start-agent starts this JaCoCo Runtime Agent.
• Whenever a class is loaded, JaCoCo can instrument the class so that it can view when the class is called and what lines (of code) are called during the testing process. By keeping this track, it builds up the code coverage statistics which is done on the fly during the second execution (i.e. generate-report).
• By default, the file is created as soon as the JVM terminates but it is also possible to run the agent in server mode. This triggers a dump of the results and the report is created before the termination. Shown below are the internals of the JaCoCo plugin:

• You can define goals and rules in the configuration of the JaCoCo-Maven plugin. This offers you the flexibility to set limits and helps in checking the amount of code coverage.
• Maven-surefire plugin is the default Maven plugin. This runs the tests in JVM and provides coverage reports. While the JaCoCo plugin instruments the code already executed by a plugin (e.g. Surefire plugin). Thus, it is a good practice to check for the dependency of the maven-surefire plugin.

Read – Getting Started With Maven for Selenium

Why is Jacoco Maven Plugin good for code coverage?

The JaCoCo-Maven plugin is appropriate for code coverage because of the following reasons:

• While working on any project, developers mostly prefer IDEs because it simplifies the coding and testing experience. JaCoCo can be installed on Eclipse IDE in the name of EclEmma, by downloading EclEmma from its marketplace.
• It is easy to add the JaCoCo plugin to all types of builds, including ANT, Maven, and Gradle. It can also be integrated with CI/CD tools like Jenkins, Circle CI, etc. This makes it versatile for a lot of use cases.
• The code coverage report generated by JaCoCo is a simple and informative HTML file that can be viewed in any browser or IDE.
• JaCoCo also provides offline instrumentation (i.e., all the classes are instrumented before running any tests).
• Analysis of a report is also quite easy, as it is color-based and provides the exact percentage of code coverage.

Code Coverage Reports Using Maven and JaCoCo Plugin

In this section, we would demonstrate the steps to generate code coverage report using the JaCoCo Maven plugin. The demonstration is done by taking a very simple test scenario. So, let’s get started.

Also, watch this in-depth tutorial to learn how to measure code coverage in Java using the JaCoCo plugin.

Pre-requisites:

• Maven 3.0 or higher: Maven is a project development management and comprehension tool. You can install the latest version from the official Apache Maven website.
• Java 1.5 or higher: You need to ensure that Java is installed on your machine. In case you do not have Java installed, please download the latest version of Java from the official Java website.
• Eclipse IDE for Java Developers: Though you can use the IDE of your choice, for this demo we have used the Eclipse IDE.

Steps to create a simple Maven Project:

• In Eclipse IDE, Go to File>New>Maven Project.

• A new dialog box appears. Make sure that the ‘Use default Workspace location’ checkbox is ticked. Click Next.

• For selecting Archetype in the project, type org.apache.maven in the textbox that is located next to the Filter. Select maven-archetype-quickstart and click Next.

• Now, specify the Group Id as com.example and Artifact Id as jacoco-example. The artifact Id is our project name. Finally, click on the Finish button.

• You can see the project file and folder hierarchy in Project Explorer.

How to specify JaCoCo Maven plugin in POM.xml

• Open POM.xml, scroll to tag. We will specify the JaCoCo-Maven plugin in the section that lists the Maven plugins. Just copy the below code and paste it above the tag.

org.jacoco
jacoco-maven-plugin
0.8.6
   <!--first execution : for preparing JaCoCo runtime agent-->
    	prepare-agent
          		prepare-agent
   <!--second execution : for creating code coverage reports-->
          report
          test
             	report

• Since we are demonstrating report generation for automated web testing with JUnit, we will also declare the JUnit dependency in POM.xml.Here is the entire content of POM.xml:

<!--?xml version="1.0" encoding="UTF-8"?-->
  4.0.0
  com.example
  jacoco-example
  0.0.1-SNAPSHOT
 jacoco-example
    UTF-8
    1.7
    1.7
<!-- JUnit dependencies added to run test cases -->
      junit
      junit
      4.11
      test
        <!-- Maven plugin for Project Management -->
          maven-clean-plugin
          3.0.0
        <!-- see http://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
        
          maven-resources-plugin
          3.0.2
          maven-compiler-plugin
          3.7.0
          maven-surefire-plugin
          2.19.1
          maven-jar-plugin
          3.0.2
          maven-install-plugin
          2.5.2
          maven-deploy-plugin
          2.8.2
                org.jacoco
                jacoco-maven-plugin
                0.8.6     
                        prepare-agent
                        prepare-agent
                        report
                        test
                        report

• From the project directory, traverse to com.example.jacoco_lambdatest package existing in src/main/java. Create a new Java class named TestMu AI.java. We will write a simple setUp() function in it that provides the desired capabilities of Selenium Grid.

package com.example.Jacoco_lambdatest;
import java.net.MalformedURLException;
import java.net.URL;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;

public class LambdaTest {
	public static String username = "";
	public static String accessKey = "";
	
  public static DesiredCapabilities setUp() throws Exception {
     DesiredCapabilities capabilities = new DesiredCapabilities();
      capabilities.setCapability("platform", "Windows 10");
	     capabilities.setCapability("browserName", "Chrome");
	     capabilities.setCapability("version", "87.0"); // If this cap isn't specified, it will just get the any available one
      capabilities.setCapability("resolution","1024x768");
      capabilities.setCapability("build", "First Test");
      capabilities.setCapability("name", "Sample Test");
      capabilities.setCapability("network", true); // To enable network logs
      capabilities.setCapability("visual", true); // To enable step by step screenshot
      capabilities.setCapability("video", true); // To enable video recording
      capabilities.setCapability("console", true); // To capture console logs
      return capabilities;
  }
}

Adding JUnit test Cases in the project:

• We will create a simple JUnit test case in AppTest.java. This is provided by default, in the src/test/java under the package name com.example.jacoco_lambdatest.

package com.example.Jacoco_lambdatest;
import java.net.MalformedURLException;
import java.net.URL;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import com.example.Jacoco_lambdatest.*;

public class AppTest {
	public static RemoteWebDriver driver;
	
	@Test
	public void testScript1() throws Exception {
		try {
	DesiredCapabilities capabilities = LambdaTest.setUp();
	String username =LambdaTest.username;
	String accessKey = LambdaTest.accessKey;
	RemoteWebDriver driver = new RemoteWebDriver(new URL("https://"+username+":"+accessKey+"@hub.lambdatest.com/wd/hub"),capabilities);       driver.get("https://lambdatest.github.io/sample-todo-app/");
     driver.findElement(By.name("li1")).click();
	driver.findElement(By.name("li2")).click();		      driver.findElement(By.id("sampletodotext")).clear();		driver.findElement(By.id("sampletodotext")).sendKeys("Yey, Let's add it to list");
	driver.findElement(By.id("addbutton")).click();
	driver.quit();					
	} catch (Exception e) {
		System.out.println(e.getMessage());
		}
	}
}

Generating code coverage reports:

• Just click on the Run As button and set the configuration as Maven Test.

• Instead, you can open cmd(Command Line), traverse to the project folder, and run the maven command, “mvn test.”

• Running the JUnit tests will automatically set the JaCoCo agent in motion. It will create a report in binary format in the target directory, with path target/jacoco.exec. The output of jacoco.exec cannot be interpreted single-handedly but other tools like SonarQube and plugins can interpret it. As we have earlier specified the jacoco:report goal, it will generate readable code coverage reports in popular formats like HTML, CSV, and XML.
• As the build is successful, go to the target folder, then to the site>jacoco folder. The code coverage report (i.e. index.html) is located in target/site/jacoco/index.html. The report looks like this:

• You can drill down at a micro level by clicking on com.example.jacoco_lambdatest>TestMu AI in the report.

• By clicking on specific functions, you will have a more detailed view in TestMu AI.java.

• Here, you would see a lot of diamonds of different colors like green, yellow and red. These are the specifications used in the report to symbolize which line of code was executed and when it was executed. We will learn more about it in the next section of the report analysis. With this, you have successfully generated a code coverage report via the Jacoco Maven plugin.

Austin Siewert

CEO, Vercel

Discovered @TestMu AI yesterday. Best browser testing tool I've found for my use case. Great pricing model for the limited testing I do 👏

2M+ Devs and QAs rely on TestMu AI

Deliver immersive digital experiences with Next-Generation Mobile Apps and Cross Browser Testing Cloud

Sign up with googleSign up with Email

Analysis of the Code Coverage Report

Our code coverage report shows 94% instruction coverage, 100% branch coverage, which is a great code coverage score. Later, we will try to achieve a 100% code coverage score by adding more test cases.

The 38 instructions shown by JaCoCo in the report refer to the bytecode instructions instead of Java code instructions. The JaCoCo reports help you visually analyze code coverage by using diamonds with colors for branches and background highlight colors for lines. A brief explanation of the diamonds seen in the code coverage report is below:

• Red diamond indicates that no branches have been exercised during the testing phase
• Yellow diamond indicates that the code is partially covered (i.e. some branches are not exercised)
• Green diamond indicates that all branches are exercised during the test

The same color code applies to the background highlight color for the line coverage. The report mainly provides three crucial metrics :

• Line coverage: This reflects the amount of code exercised based on the number of Java byte code instructions called by the tests.
• Branch coverage: This shows the percentage of exercised branches in the source code. These are typical if/else or switch statements.
• Cyclomatic complexity: This reflects code complexity via the number of paths needed to cover all possible paths in a code. It also refers to the number of test cases needed to implement to cover the entire code. As there is no switch or statement in the code, the cyclomatic complexity will be 1; only one execution path is sufficient to cover the entire code.

Introducing more test cases for improving code coverage

• To achieve better code coverage, more tests need to be introduced that test the code that was not covered earlier via the test implementation.
• Go to AppTest.java in src/test/java to add more test cases.
• The new test cases added to AppTest.java will look as follows:

@Test
	public void testScript2() throws Exception {
	    try {
	    DesiredCapabilities capabilities = LambdaTest.setUp();
	    String username = LambdaTest.username;
	    String accessKey = LambdaTest.accessKey;
	    RemoteWebDriver driver = new RemoteWebDriver(new URL("https://"+username+":"+accessKey+"@hub.lambdatest.com/wd/hub"),capabilities);		 driver.get("https://lambdatest.github.io/sample-todo-app/");		    driver.findElement(By.name("li2")).click();
	    driver.findElement(By.name("li3")).click();
	   driver.findElement(By.id("sampletodotext")).clear();	driver.findElement(By.id("sampletodotext")).sendKeys("Yes, Let's add it to list");				driver.findElement(By.id("addbutton")).click();
					driver.quit();					
		} catch (Exception e) {
				System.out.println(e.getMessage());
			    }
			}
	
	@Test
	public void testScript3() throws Exception {
		try {
	    DesiredCapabilities capabilities = LambdaTest.setUp();
	    String username = LambdaTest.username;
	    String accessKey = LambdaTest.accessKey;
	    RemoteWebDriver driver = new RemoteWebDriver(new URL("https://"+username+":"+accessKey+"@hub.lambdatest.com/wd/hub"),capabilities);			  driver.get("https://lambdatest.github.io/sample-todo-app/");		    driver.findElement(By.name("li4")).click();
	    driver.findElement(By.id("sampletodotext")).clear();	driver.findElement(By.id("sampletodotext")).sendKeys("Yes, Let's add  it!");
	  driver.findElement(By.id("addbutton")).click();
	  driver.quit();					
	} catch (Exception e) {
		System.out.println(e.getMessage());
			}
			}

• Let’s run the maven jacoco:report to publish a new coverage report.

• JaCoCo offers a simple and easy way to track code coverage score by declaring minimum requirements. Build fails if these requirements are not met else the build is successful.
• These requirements can be specified as rules in POM.xml. Just specify the new execution specifying ‘check’ goal in POM.xml. Add the below code after the second <execution> tag in POM.xml.

<!--Third execution : used to put a check on the entire package-->
    jacoco-check
        check            
                PACKAGE       
                        LINE
                        COVEREDRATIO
                        0.50

• With this, we are limiting our coverage ratio to 50%. This signifies a minimum of 50% of the code should be covered during the test phase.
• You can run maven clean verify to check whether the rules set in jacoco:check goal are met or not.

• The log shows “All coverage checks have been met.” as our code coverage score is 94% which is greater than our minimum 50%.

Automation Testing On TestMu AI Selenium Grid using Maven Project with Jacoco Plugin

Selenium testing on the cloud helps you attain better browser coverage, increased test coverage, and accelerated time to market. Parallel testing in Selenium helps you achieve the above mentioned requirements.

TestMu AI Cloud Selenium Grid is a cloud-based scalable Selenium testing platform that enables you to run your automation scripts on 2000+ different browsers and operating systems.

Pre-requisites:

To run the test script using JUnit with Selenium, first, we need to set up an environment. You would first need to create an account on TestMu AI. Do make a note of the username and access-key that is available in TestMu AI profile section.

We will use this sample project for Java Selenium testing.

Importing project to Eclipse IDE:

After downloading a zip file of the project: junit-selenium-sample from GitHub, we import it to Eclipse IDE by following the below mentioned steps:

• Go to your Eclipse IDE, click on the File menu and select Import. A new dialog box appears.
• Type Maven in the textbox below and select Existing Maven Projects, and then click Next.

• In the next dialog box, click on Browse and traverse to the project folder. Also, tick the checkbox giving the path to the POM.xml file, and click Finish.

• Your project will be loaded in Eclipse IDE successfully.

Adding dependencies in the POM.xml file:

• Open the POM.xml, now add the dependencies of JUnit, Selenium, and JaCoCo Maven Plugin. After adding the dependencies to the code of POM.xml should look like this:

<!--?xml version="1.0" encoding="UTF-8"?-->
    4.0.0
    com.lambdatest
    lambdatest-junit-sample
    1.0-SNAPSHOT
     UTF-8
        2.19.1
        default
        <!--JUnit dependency-->
            junit
            junit
            4.12
            test
            commons-io
            commons-io
            1.3.2
            test
        <!--Selenium dependency-->
            org.seleniumhq.selenium
            selenium-java
            2.52.0
            test      
            com.googlecode.json-simple
            json-simple
            1.1.1
            test
            <!--Apache Maven Plugins-->
                maven-compiler-plugin
                3.0
                    1.8
                    1.8
                maven-surefire-plugin
                2.19.1
            <!--JaCoCo Maven Plugin-->
                org.jacoco
                jacoco-maven-plugin
                0.8.6      
                        prepare-agent
                      prepare-agent
                        report
                        test
                        report

<em><a href="https://github.com/rachnaagrawal/junit-selenium-sample/blob/master/pom.xml" target="_blank" rel="nofollow noopener">Github</a></em>

Configuring the Desired Capabilities for JUnit Automation Testing:

• To connect to TestMu AI Selenium Automation Grid, the first thing done is invoking a remote webdriver. This remote driver requires some capabilities like browser, browser versions, operating system, etc., to build an environment. The code to it looks as follows:

WebDriver driver = new RemoteWebDriver(new URL("https://" + username + ":" + accesskey + "@hub.lambdatest.com/wd/hub"),
DesiredCapabilities.firefox()); //A class named DesiredCapabilities is used to create an environment as a Firefox browser.

• In the JUnit automation testing code, the capabilities like browser, browser versions, operating system information, etc., can be customized and are passed via Capabilities Object.
• TestMu AI has made this process very easy by providing an inbuilt Capabilities Generator. The Capabilities Generator will automatically generate the code for desired capabilities based on your input. Such as, our configurations are:

Fields	Selected Values
Operating Systems	Windows 10
Browser	Chrome
Browser Version	62.0
Resolution	1024×768
Selenium Version	3.13.0

• Selecting the above-specified configuration in the Capabilities generator and paste it to TestMu AIBseTest.java.

Specifying TestMu AI Username and Access Key in required Java Class:

• In the Project Explorer, you would see 3 Java classes: TestMu AIBaseTest.java (contains the setup required for Java Selenium testing), Parallelized.java (contains Junit tests for parallel testing on TestMu AI Selenium grid), and SimpleTest.java (contains simple unit tests).

• TestMu AIBaseTest.java fetches the required data like desired capabilities, username, and access key from a config file. This is provided in src/test/resources as config.json.
• Specify the desired capabilities, Username, and Access key in config.json. This JSON file is used as you can provide multiple configurations in it for realizing parallel testing with Selenium. to specify multiple configurations in config.json and then fetch them later.

[ { "tunnelName":"LambdaTest tunnel", 
"buildName":"running Java Selenium Tests",
"testName":"Jacoco JUnit Test",
"username": "user-name",
"access_key":"access-key", 
 "operatingSystem" : "win10", 
 "browserName" : "chrome",  
 "browserVersion" : "62.0", 
 "resolution" : "1024x768" }]

Unit Testing using JUnit with Selenium:

• SimpleTest.java is the Java class for specifying a single unit test case for testing and performing code coverage using the JaCoCo Maven plugin.

package com.lambdatest;

import com.lambdatest.LambdaTestBaseTest;
import org.junit.Test;
import org.openqa.selenium.By;
import static org.junit.Assert.assertEquals;

public class SimpleTest extends LambdaTestBaseTest {
    /**
     * Simple Test case annotation for JUnit Test
     * @throws Exception
     */

   @Test
    public void validateUser() throws Exception {
    	  driver.get("https://lambdatest.github.io/sample-todo-app/");
	  driver.findElement(By.name("li1")).click();
	  driver.findElement(By.name("li2")).click();
	  driver.findElement(By.id("sampletodotext")).clear();
	  driver.findElement(By.id("sampletodotext")).sendKeys("Yey, Let's add it to list");
	  driver.findElement(By.id("addbutton")).click();
    }

}

• This is a simple Selenium WebDriver test that will open a sample to-do application that will do the following task:
  • Mark the first two items as mark done.
  • Add a new item to the list.
  • Return the added item.
• Trigger the command mvn test on the terminal to build and run the test case.

• Now, login to your TestMu AI Account and go to Automation. You will find the tests you ran under the build name “JUnit JaCoCo Tests.”

• Click on “JUnit JaCoCo Tests” and review them in detail. TestMu AI has recorded the video. So you can see the visuals too.

Generating Code Coverage Report via JaCoCo Maven Plugin:

• Now, as we have run the JUnit test cases on TestMu AI Selenium Grid, the code coverage report should be generated via the JaCoCo Maven plugin.
• Just go to the target folder, you will find the binary format of the report as jacoco.exec.
• You can view the HTML, CSV, and XML form of the code coverage report in the target/site/jacoco folder as index.html, jacoco.csv, and jacoco.xml, respectively.
• Great! Now you can also try to improve the code coverage score and analyze the code coverage report generated.