Fault-Based Testing and the Pesticide Paradox

Mutation testing

As mentioned, mutation testing is the most common form of fault-based testing, in which by slightly modifying the original code we create several mutants. A reliable test data set should then differentiate the original code from the well-selected mutants. In mutation testing, we introduce faults into the code to see the reliability of our test design. Therefore, mutation testing is essentially not testing, but “testing the tests”. A reliable test data set has to “kill” all of them. A test kills a mutant if the original code and the mutant behave differently. For example, if the code is y = x and the mutant is y = 2 * x, then a test case x = 0 does not kill the mutant while x = 1 does.

If a mutant hasn’t been killed, then the reasons can be the following:

• Our test case set is not good enough and we should add a test that will kill the mutant.
• The mutant is equivalent to the original code. It’s not an easy task to decide the equivalence. And what is more, the problem is undecidable in the worst case.

In the case of a first-order mutant, the code is modified in one place. In the case of a second-order mutant, the code is modified in two places, and during the execution, both modifications will be executed. Offutt showed that the coupling effect holds for first and second-order mutants, i.e., only a very small percentage of second-order mutants were not killed when all the first-order mutants were killed. Moreover, Offutt showed that the second-order mutant killing efficiency is between 99.94% and 99.99%. This means that if we have a test set that will kill all the first-order mutants, then it will also kill the second-order mutants by 99.94-99.99%. Thus, it is enough to consider only simple mutants, which is much easier, and there are much fewer of them.

The real advantage is that if we have a test design technique that kills the first-order mutants, then this technique kills the second and higher-order mutants as well. Anyway, we can assume that the bigger the difference between the correct and the incorrect code is, the higher the possibility is to find the bug.

Excellent, we have a very strong hypothesis that if we have a good test design technique to find the first-order mutants as bugs, we can find almost all the bugs, and our software becomes very high quality. A test design technique that finds at least 99.94% of the bugs leaves at most 6 of every 10,000 faults undetected.

Ok, we made mutation testing and we found that 95% of the mutants have been killed. What’s next? The first solution is to create the missing test cases. We should consider the basic code and a non-killed mutant. We should find an input for which the mutant is killed. This is our new test case. Here the method is reversed concerning testing. We start from the faulty code, the mutant, and search for a test that reveals this bug. If we cannot find a test though we know the fault, then we can assume that a user without this knowledge will not detect the bug either. In other words, we consider this mutant as equivalent even if we cannot prove it. If the tester hasn’t killed some mutants, then instead of the non-killed mutants, the missing test cases appeared. Knowing your missing tests, you can improve your test design knowledge.

Sometimes, finding the missing test cases for the non-killed mutants is too costly. Unfortunately, knowing that we killed, for example, 95% of the mutants doesn’t mean that our test case efficiency is 95% (i.e., that we can find 95% of the bugs in the code). Generating mutants (bugs) is quite different from making mistakes by the developers. However, historical data can help. If in the past our mutant killing efficiency was 95%, the code quality was good enough for a given risk level, and now it’s only 80% for the same risk, then our test case set is not good enough. In this case, we should find additional test cases based on the mutants until we reach 95%.

We note that fault-based testing differs from defect-based testing since the latter is a test technique in which test cases are developed from what is known about a specific defect type (see ISTQB glossary).

How Are LLMs Changing Mutation Testing?

Large language models (LLMs) now generate both the mutants and the tests that kill them. That cuts the two costs described above: too many mutants to execute, and the manual work of finding a test for every surviving mutant.

Meta's ACH system is a published example. The Mutation-Guided LLM-based Test Generation at Meta paper reports these results:

• Targeted mutants: ACH generates relatively few mutants compared with traditional mutation testing. Each mutant simulates a fault tied to a specific concern, such as privacy.
• Scale of the run: Applied to 10,795 Android Kotlin classes across 7 Meta platforms, ACH produced 9,095 mutants and 571 privacy-hardening test cases.
• Equivalent mutant detection: An LLM agent flags mutants that behave like the original code, the undecidable case listed above. The agent reached 0.79 precision and 0.47 recall, rising to 0.95 and 0.96 with simple pre-processing.
• Engineer acceptance: Engineers who reviewed the tests for Messenger and WhatsApp accepted 73% of them and judged 36% privacy relevant.

A generated test faces the same check as a hand-written one. In unit testing, keep a generated test only when it kills a mutant that the existing tests miss. Review it before it joins a regression testing suite, because an LLM can write a test that passes without asserting anything useful.

Pesticide paradox

Most testers know and believe in ‘the 7 principles of software testing’. One of them is the pesticide paradox. Originally, Beizer (1990) wrote: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.” Now this paradox is described as ‘If the same set of repetitive tests is conducted, the method will be useless for discovering new defects.’ or ‘if the same set of test cases are executed again and again over the period of time then these set of tests are not capable enough to identify new defects in the system.’ In simple words: tests wear out.

Let us assume that we have a good test case set that kills all the first-order mutants. This is not entirely perfect as some bugs haven’t been found (0,06%). Assume that only one bug cannot be found with our tests. Now assume that the specification remains but the code is refactored several times. Except for the only bug, all the others made during the code modification will be detected. Let us assume again that the test case set doesn’t kill a single mutant only. The argument is the same. Any faulty code modification will be detected except for these two bugs.

Therefore, we can say that the error-revealing capacity of a test set is determined by the mutation-killing efficiency that is constant until the requirements remain unchanged. This is independent of whether the mutation testing is done or just could have been done. Therefore, if the cause of the new defects is faulty refactoring, and the requirements remain unchanged, then the good test set remains as good for the modified code as for the original. On the other hand, when the requirements change, or new requirements are added, then new/modified tests must be written. But it does not mean that the original tests wear out; on the contrary, we use (some of) them in our regression test pack. The conclusion is that the pesticide paradox principle is not valid.

Reference

Beizer, B (1990)., Software Testing Techniques, 2nd Edition Itp – Media, ISBN-10 1850328803