Code Coverage vs Test Coverage: Which Is Better?

How To Perform Code Coverage?

There are different levels at which the code coverage is possible, some of the common subtypes of code coverage are:

• Branch coverage – Branch coverage also called decision coverage is used to ensure that every possible branch used in a decision-making process is executed. For example, if you are incorporating a fall back for cross browser compatibility using an If…An else conditional statement or by a Do…While statement in your code, as a part of coverage; you have to ensure that all the branches i.e. If, Else, Do, and While are tested by providing appropriate inputs to make a cross browser compatible website.
• Function coverage – Function coverage ensures that necessary functions (especially exported functions/APIs) are tested. This should also include testing the functions with different types of input parameters as that would also test the logic used in the functions. Once all the functions in the code are tested, function coverage would be 100%.
• Statement Coverage – This is an important code coverage methodology in which test code has to be written in a manner that every executable statement in the source code is executed at least once. This also includes corner cases or boundary cases.
• Loop Coverage – This approach is to ensure that every loop in the source is executed at least once. There might be some loops that may execute based on results that you achieved at runtime, it is important to test such loops as well in order to make the code foolproof.

For checking the code coverage, an approach called instrumentation is used. Instrumentation can be used to monitor the performance, insert trace information, and diagnose any kind of errors in the source code. There are different kinds of instrumentation and there could be a minimal performance (and timing) overhead depending on the instrumentation approach being used.

Types of Instrumentation

There are three major types of instrumentations

• Code instrumentation – Here the source code is compiled after the addition of instrumentation statements. The compilation should be done using the normal toolchain, successful compilation results in the generation of instrumented assembly. For example, in order to check the time taken to execute a particular function in your code, you can add instrumentation statements in Start & End of the function.
• Runtime instrumentation – Contrary to the code instrumentation approach, here the information is collected from the runtime environment i.e. when the code is under execution.
• Intermediate code instrumentation – In this type of instrumentation, an instrumented class is generated by the addition of byte codes to the compiled class files.

Depending on your test requirement, you should choose the right code coverage tool and the best instrumentation approach supported by the tool.

Tools for Code Coverage

There are a number of code coverage tools supporting different programming languages and many of them double up as QA tools as well. Many of the tools can be integrated with build tools and project management tools which make them much more powerful & useful. While choosing an open-source code coverage tool, you should check the features supported by the tool and whether there is an active development of the tool. Based on those factors, below are some of the popular open-source code coverage tools.

• Coverage.py – It is a code coverage tool for Python. As the name suggests, it analyzes your source code and identifies the percentage of code that was executed. It is developed in Python.
• Serenity BDD – Supporting Java & Groovy programming languages, Serenity BDD is a popular open-source library that is primarily used for writing excellent quality acceptance tests faster. You can make use of stories & epics for the tests and the code coverage is computed for these stories & epics. Due to this, the test reports that are generated are more illustrative and narrative in nature. You can map those automated tests back in your requirements.

  It can be used with JUnit, Cucumber, and JBehave. Serenity BDD can be easily integrated with Maven, Cradle, JIRA, and Ant. If you are using the Selenium WebDriver or Selenium Grid framework for automation testing, choosing Serenity BDD can be a huge advantage due to its compatibility with Selenium. You can refer https://www.thucydides.info/#/documentation for further information.

• JaCoCo – JaCoco is a code coverage tool for Java. Though there are other options like Cobertura & EMMA, these tools were deprecated since there was no update for a long time. The JaCoCo tool is a part of the Eclipse Foundation and it replaced the EMMA code coverage tool in Eclipse. Apart from the active development of JaCoCo, another advantage of using it is seamless integration with CI/CD & project management tools like Maven, Jenkins, Gradle, etc. Please visit official site for more information.
• JCov – JCov is a test framework agnostic code coverage tool. It can be effortlessly integrated with Oracle’s test infrastructure – JavaTest & JTReg. Though it is not in active development, support for on-the-fly instrumentation & offline instrumentation are the major advantages of using JCov. Please visit https://wiki.openjdk.java.net/display/CodeTools/jcov for further information.
• PITest – Most of the code coverage tools check the code for branch coverage, statement coverage, loop coverage, etc. and give out the coverage results. Though that information is useful to improve the quality of your test code, the test does not tell you how good the automation tests were in discovering bugs. This is where mutation testing can be very helpful.

  PITest is a very popular code coverage tool that is used for mutation testing for Java & JVM. It does the job of mutation testing by modifying your test code and the unit tests are now performed on this modified code. If bugs are discovered using this code i.e. after extra code is added by PITest, the unit test is foolproof; else it needs changes as the issues were left undiscovered. PITest is easy to use, fast and is under active development. It also integrates with popular CI/CD tools which makes it much more useful. Please refer http://pitest.org/ for more information.

Understanding Test Coverage In-Depth

Unlike code coverage which is a white-box testing methodology, test coverage is a black-box testing methodology. Test cases are written in a manner that there is maximum coverage of requirements mentioned in FRS (Functional Requirements Specification), SRS (Software Requirements Specification), URS (User Requirement Specification), etc. Since the tests are derived from these documents, there are minimal/no chances of automation.

How To Perform Test Coverage?

Like Code coverage, Test coverage can also be evaluated through different types of testing. However, which type of testing should you follow, depends entirely on your business proposition For example – In user-centric web applications, there might be a case where the UI/UX tests are of higher priority over functional tests whereas in other type of applications (e.g. banking, finance); usability tests, security tests, etc. might be more important.

Some of the test coverage mechanisms are below:

• Unit Testing – This type of testing is performed at a unit level/module level. Bugs encountered at a unit level can be different from the issues encountered at the integration stage.
• Functional Testing – In functional testing, the functions/features are tested against the requirements mentioned in the Functional Requirement Specification (FRS).
• Integration Testing – It is also called as system testing since the software is tested on a system level. This type of testing is performed once all the required modules are integrated.
• Acceptance Testing – It all depends on the result of acceptance testing whether the product would be released to the end consumer/customer. This is where the developers get a sign-off as a green pass from the testers and SMEs of the web-application, right before pushing the code changes from the Staging environment to the Production.

The other important point to note is that the purpose & meaning of test coverage can differ depending on the level at which testing is performed. It also depends on the type of product on which black-box testing is performed. The test coverage metrics for testing mobile phones would differ from metrics for e-commerce website testing. Some classifications are below:

• Features coverage – Here the test cases are developed in a manner that there is maximum coverage of product features. For example, if a tester is assigned the job of testing a phone dialer application; he should make sure that the number being dialed is proper in length. If the testing is done in India, the mobile number should be maximum 10 digits; else it should flash an error. Similar to this, all the mandatory & optional features have to be tested according to a priority set by the product team.
• Risks coverage – Every product/project requirement document has a section that mentions about the Risks & Mitigation associated with the project. Though some risks (e.g. changes in business dynamics) are beyond the scope of the planning/development/test team, there are some risks that need to be addressed during the testing phase.

  For example, while developing a business website, the server infrastructure should be set up in a manner that page access is very fast. Depending on the location (i.e. country, city, etc.) from where the website is accessed, the closest server should be chosen for loading the website; else the entire experience would be hampered. The test team should also perform a load test where a performance test is done when multiple numbers of users are trying to access the website at the same time i.e. scenario where there is huge traffic on the website. If the results of these tests are not good, it can result in below-average user experience (which can be a huge risk).

• Requirements coverage – Here tests are defined in a manner that there is maximum coverage of the product requirements mentioned in various Requirement Specification documents.

  For example, if you are testing a pre-installed SMS application, you need to make sure that the default language is set properly e.g. If the mobile is marketed in a country where English is not the primary language e.g. China; the default SMS language should be Chinese and for other customers e.g. India; the default language should be English.

Tools for Test Coverage

In the case of code coverage, the measurement metric is the percentage of code that is tested through test cases/test suites. Hence, the test results can be quantified i.e. Out of 100 LOC (Lines of Code), the code coverage was 80 lines. This means that the code coverage is 80%. The results of test coverage cannot be quantified since the tests are performed to verify the functional requirements. You can also come up with black-box tests that can test more than one requirement in a single test e.g. In order to test the failure scenario in a simple email login-page, you can write a test case that enters an email address without @ symbol than try to proceed with the login. This would test the functionality of the login page and also check if the logic for verifying the format of the email address is working as per the requirement.

Though there are few instances where you have to write test code for achieving the test coverage requirement, you may still need to use some of the popular test frameworks in some cases. Two of the most popular test frameworks are:

• JUnit – JUnit is the unit testing framework for Java. It can also be used for UI testing. It is open-source and considered important in the development of TDD (Test Driven Development). Developers & testers make use of JUnit to write & execute tests that are repetitive. This also makes it a popular framework for regressive testing. Please refer https://junit.org/junit5/ for more information.

Run You First Selenium Automation Testing Script with JUnit

• PyUnit – PyUnit (also called as Python Unit Testing Framework) is a widely used testing framework primarily used for unit testing. It is the Python port of JUnit and is used by Python developers who follow the TDD approach. PyUnit supports development of test cases, test suites, test fixtures, etc. The unittest module is the core of PyUnit framework. Please refer https://docs.python.org/2/library/unittest.html for more information.

Run You First Selenium Automation Testing Script with PyUnit

Though there are many other tools/test frameworks using which developers/testers can write test code, JUnit & PyUnit are the most popular testing frameworks for their respective programming languages.

Code Coverage vs Test Coverage; Which One Should You Follow?

The bases on which the impact of code coverage and test coverage are measured are totally different. Code coverage is measured by the percentage of code that is covered during testing, whereas test coverage is measured by the features that are covered via tests.

The million-dollar question is “Which one of these is best suited for your project”?. There is no definite answer to this question as the solution depends on the type & complexity of the project. In most cases, test coverage and code coverage are used since they are equally important in software projects.

Advantages of Test Coverage

• A good approach to test the software features and compare the results across different specification documents (requirements, feature, product, UI/UX, etc.)
• As the tests performed as a part of the coverage are black-box in nature, executing those tests might not require much expertise.

Shortcomings Of Depending On Test Coverage

• There is no scope of automation as the tests are predominantly black-box tests. Manual comparison of the test results has to be done with the expected output since these tests are performed at a ‘feature level’ and not ‘code level’.
• No concrete way of measuring test coverage. Hence, the coverage results largely depend on the domain competence of the tester who is performing the tests and may vary from one tester to another.

Advantages of Code Coverage

• Provides the effectiveness of your test code and how you can improve the coverage
• Irrespective of the type of tool being used (open-source, premium), setting up a code coverage tool should not take much time.
• Helps in improving the code quality by capturing bugs in the code.

Shortcomings of Code Coverage

• Majority of code coverage tools are limited to unit testing.
• The methodology used by tools could be different hence; you may not be able to compare code coverage results of one tool to another.
• Searching for the best-suited tool could be a big task as you need to compare & try features from those tools before selecting the best one that suits your project requirements.
• There are very few tools that provide support for different programming languages e.g. Java, Python, C, etc. Hence, you may need to have more than one tool in case your team is using multiple programming languages (for test code development).