Chapter 11. Test Quality
Code coverage is often used to check the quality of tests, which is not a good idea. As we have learned in this section, coverage measures are not credible indicators of test quality. Code coverage might help by showing you deficiencies in your safety net of tests, but not much more than this. In addition to code coverage, you should use other techniques (i.e. visual inspection - see Section 11.5) to supplement its indications.
11.4. Mutation Testing
High quality software can not be done without high quality testing. Mutation testing measures how “good” our tests are by inserting faults into the program under test. Each fault generates a new program, a mutant, that is slightly different from the original. The idea is that the tests are adequate if they detect all mutants.
— Mattias Bybro A Mutation Testing Tool For Java Programs (2003)
As discussed in the previous section, code coverage is a weak indicator of test quality. Let us now take a look at another approach, called mutation testing, which promises to furnish us with more detailed information regarding the quality of tests.
Mutation testing, although it has been under investigation for decades, is still in its infancy. I discuss this topic here because I believe it has great potential, which is waiting to be discovered. Recent progress in mutation testing tools (i.e. the PIT mutation testing tool14) leads me to believe that this great and exciting idea will finally get the attention it deserves. By reading this section you might just get ahead of your times by learning about a new star that is about to outshine the existing solutions, or …waste your time on something that will never be used in real life.
Suppose you have some classes and a suite of tests. Now imagine that you introduce a change into one of your classes, for example by reverting (negating) one of the if conditional expressions. In doing this you have created a so-called mutant of the original code. What should happen now, if you run all your tests once again? Provided that the suite of tests contains a test that examines this class thoroughly, then this test should fail. If no test fails, this means your test suite is not good enough15. And that is precisely the concept of mutation testing.
11.4.1. How does it Work?
Mutation testing tools create a plethora of "mutants": that is, slightly changed versions of the original production code. Then, they run tests against each mutant. The quality of the tests is assessed by the number of mutants killed by the tests16. The tools differ mainly in the following respects:
•how the mutants are created (they can be brought to life by modifying source code or bytecode),
•the set of available mutators,
•performance (e.g. detecting equivalent mutations, so the tests are not run twice etc.).
Mutants are created by applying various mutation operators - i.e. simple syntactic or semantic transformation rules - to the production code. The most basic mutation operators introduce changes to
14http://pitest.org
15In fact, if all tests still pass, it can also mean that the "mutant" program is equivalent in behaviour to the original program. 16Okay, I admit it: this heuristic sounds like it was taken from the game series Fallout: "the more dead mutants, the better" :).
245
Chapter 11. Test Quality
the various language operators - mathematical (e.g. +, -, *, /), relational (e.g. =, !=, <, >) or logical (e.g. &, |, !). An example of a mutation would be to switch the sign < to > within some logical condition. These simple mutators mimic typical sources of errors - typos or instances of the wrong logical operators being used. Likewise, by changing some values in the code, it is easy to simulate off-by-one errors. Other possible mutations are, for example, removing method calls (possible with void methods), changing returned values, changing constant values, etc. Some tools have also experimented with more Javaspecific mutators, for example relating to Java collections.
11.4.2. Working with PIT
PIT Mutation Testing is a very fresh Java mutation testing tool, which has brought new life to the rather stagnant area of mutation testing tools. It works at the bytecode level, which means it creates mutants without touching the source code. After PIT’s execution has finished, it provides detailed information on created and killed mutants. It also creates an HTML report showing line coverage and a mutation coverage report. We will concentrate on the latter, as line coverage has already been discussed in Section 11.3.
We will use a very simple example to demonstrate PIT in action and confront it with code coverage tools. Listing 11.4 shows our "production code", which will be mutated by PIT17.
Listing 11.4. Method with two if statements
public class TwoIfs {
public int twoIfs(int a, int b) { if (a > 0) {
return 1; } else {
System.out.println();
}
if (b > 0) { return 3;
} else { return 4;
}
}
}
Let us say that we also have a test class which (supposedly) verifies the correctness of the twoIfs() method:
Listing 11.5. Tests for the twoIfs method
public class TwoIfsTest {
@Test
public void testTwoIfs() { TwoIfs twoIfs = new TwoIfs();
assertEquals(1, twoIfs.twoIfs(1, -1)); assertEquals(3, twoIfs.twoIfs(-1, 1)); assertEquals(4, twoIfs.twoIfs(-1, -1));
}
}
17The idea of this code is taken from the StackOverflow discussion about code coverage pitfalls - http://stackoverflow.com/ questions/695811/pitfalls-of-code-coverage.
246
Chapter 11. Test Quality
What is really interesting is that this simple test is good enough to satisfy the code coverage tool - it achieves 100% in respect of both line and branch coverage! Figure 11.4 shows this:
Figure 11.4. 100% code coverage - isn’t that great?
When we execute a PIT analysis, it will create mutants of the production code from Listing 11.4 by reverting the inequality operators and fiddling with comparison values. Then it will run all tests (in our case only the one test shown in Listing 11.5) against each mutant and check if they failed.
The outcome report shows the code that was mutated together with some information about applied mutations. Just like with code coverage reports, the red background denotes "bad" lines of code, which means that some mutations performed on these lines went unnoticed when testing. Below the source code there is a list of applied mutations. From this list we can learn that, for example, one mutant survived the change of conditional boundary in line 6. The "greater than" symbol was changed to "greater or equal" and the tests still passed. The report informs us that such and such a mutant SURVIVED, which means it was not detected by our tests.
247
Chapter 11. Test Quality
Figure 11.5. Mutation testing - PIT report
This simple example shows the difference between code coverage and mutation testing: in short, it is much simpler to satisfy coverage tools, whereas mutation testing tools can detect more holes within your tests.
11.4.3. Conclusions
Mutation testing has been around since the late 1970s but is rarely used outside academia. Executing a huge number of mutants and finding equivalent mutants has been too expensive for practical use.
— Mattias Bybro A Mutation Testing Tool For Java Programs (2003)
Mutation testing looks interesting, but I have never once heard tell of it being used successfully in a commercial project. There could be many reasons why this idea has never made it into developers' toolboxes, but I think the main one is that for a very long time there were no mutation testing tools that were production-ready. Existing tools were lagging behind relative to the progress of Java language (e.g. not supporting Java 5 annotations), and/or were not up to the industry standards and developers’ expectations in terms of reliability and ease of use. Because of this, code coverage, which has had decent tools for years, is today a standard part of every development process, while mutation testing is nowhere to be found. As for today, developers in general not only know nothing about such tools, but are even unaware of the very concept of mutation testing!
This situation is likely to change with the rise of the PIT framework, which offers much higher usability and reliability than any other mutation testing tool so far. But it will surely take time for the whole
248