Chapter 11. Test Quality

Listing 11.2. Tests of the Money class

public class MoneyTest {

@Test

public void constructorShouldSetAmountAndCurrency() { Money money = new Money(10, "USD");

assertEquals(10, money.getAmount()); assertEquals("USD", money.getCurrency());

}

@Test

public void shouldBeAbleToAddMoney() { assertEquals(new Money(7, "USD"),

new Money(3, "USD").add(new Money(4, "USD")));

}

}

The tests shown in Listing 11.2 are not the best we have seen so far in terms of how they are written, but this is not the point. The thing is, they are highly incomplete, given the requirements that the Money class is supposed to fulfill. We have already discussed the notion that each feature should be tested with more than one test case (i.e. with many values - see Section 6.1). However, it seems that one can obtain quite high coverage by testing just some features with a single test case.

This is an interesting observation. Interesting, or maybe rather ominous, considering the number of people who feel entitled to an opinion regarding the quality of tests based on code coverage measurements.

What we could do, to achieve even higher coverage, would be to add some more tests. Listing 11.3 shows such an attempt. This test was created only in order to make the branch code coverage higher. If we measure the branch coverage now, we will learn that it has gone up from 50% to 75%.

Listing 11.3. Additional test method of the Money class

public void differentMoneyShouldNotBeEqual() {

assertFalse(new Money(7, "CHF").equals(new Money(7, "USD"))); assertFalse(new Money(8, "USD").equals(new Money(7, "USD")));

}

Please note that there are many important features of the Money class that we are still not verifying. For example, it would be nice to make sure that objects of the Money class are immutable, but if you think about such a test, it would not make the code coverage any higher. We would also like to make sure the addition really works, by verifying it with some more examples. But no, we have not done anything like this, and yet… voila! - we already have 86% line coverage and 75% branch coverage.

This simple example reveals a very important weakness of code coverage measures. Code coverage tools do not measure whether tests cover requirements! They only measure what parts of the production code were executed when running the tests.

11.3.4. How Much Code Coverage is Good Enough?

You should, at least once in your lifetime, get 100% coverage – just so you know how it tastes. Then you can decide whether it is worth the effort.

— Jaroslav Tulach Geecon 2011 Talk (paraphrased)

242

Chapter 11. Test Quality

I can get 100% code coverage and test nothing because I have no asserts. Stop making coverage a goal, it’s only a way!

— Twitter @unclebobmartin 2011 Apr 8

Since code coverage exists, there must also be a question we can ask about what is the good/appropriate/ required amount of code coverage. The question appears in many forms, usually as a "should I aim at 100% coverage"? Like many other dilemmas, this one has also received some quite contradictory answers. Let us have a look at this.

From the examples presented so far, we should already understand that high code coverage, even 100% code coverage, does not mean the tests are of high quality. You can write tests of no value at all (e.g. tests without assertions), whose only goal is to prop up the coverage metrics. In fact, this often happens when management demands that developers obtain certain, usually very high, values of code coverage. If the team cannot fulfill these requirements (for example because of lack of skills), the coverage metrics will be meaningless. Their only value will be in keeping management happy. Nice in the short term, but not very useful in the long run.

The use of red and green colors makes coverage reports resemble those of tests results. On the one hand, this is convenient, because the "green-is-good" and "red-is-bad" associations are probably well entrenched in developers’ minds. On the other hand, a red color used in a test result, and the same red color in a coverage report, do not mean the same thing. Yet it makes developers follow a "no red policy" and focus on eliminating anything that shows up as red. This means they might try to get 100% coverage even though, as we have already noted, this might not be the most important thing needing to be done.

This raises another issue. High coverage measures, together with the misguided assumption that "tests ensure that it works", might lead you to a (false) feeling of security. However, in the light of what was discussed previously, you should not feel secure, even if your code coverage does reach the 100% mark.

"Our code is working fine because we have 100% code coverage" is a fallacy.

Also, with regard to what was said earlier in connection with "things too simple to break" (see Section 10.5), there are some parts of the code which do not deserve to be tested. This makes 100% code coverage a goal not worth chasing, as it makes developers write tests which are not really worth the effort. A grotesque example might be writing a test for a private constructor in a static class, which is there for the sole purpose of never being called…

So, what is the answer to the question which opened this section? How much coverage is desirable, and how much should be considered inadequate? A popular answer to this has been given by [savoia2007]. Basically, it suggests the following13:

•If you begin your adventure with testing, you should not worry about code coverage. Concentrate on writing good tests and honing your testing skills. If you spend too much time thinking about coverage, you will get depressed (because it is unlikely that you will be able to achieve high values at this point).

•If you are experienced, then you know that there are no simple answers, and you are able to handle this truth and keep on working. No threshold of required code coverage can be taken as given, because the desired level depends on many factors that only you, the author of the code, can possibly understand.

What I, myself, would add, is that no matter what level of experience you have, you should focus on testing all of the important requirements of your classes, and only then check code coverage.

13I would encourage you to read the original version of the article.

243

Chapter 11. Test Quality

11.3.5. Conclusion

Code coverage tells you what you definitely haven’t tested, not what you have.

— Mark Simpson StackOverflow discussion 2009

In this section we have seen the two faces of code coverage. The first face is friendly and helpful. It allows you to get a good understanding of the parts of the code which are not covered by tests and thus find the gaps in your safety net. However, there is also another face to code coverage. This one makes you worry about seeing the color red on coverage reports and waste time writing tests for lines that are not worth it. It also makes you feel secure, and makes you believe that your code works as required (when neither of these are in fact justified).

The point is, that you should choose what is good in code coverage and ignore the rest. You should use it to:

•Learn what you definitely have not tested.

•Make sure you do not lose some tests during huge redesigns of your test code (which happens rarely).

•Have a broader picture of gaps in your safety net of tests.

•This is especially useful if you want to see the time trend of coverage and compare it with other things that happened within the team (trainings, new people joining the team, technology change, etc.)

•It is also good for seeing if there are some parts of the code which are typically being undertested. For example, team members might not have the skills needed for testing expected exceptions, thus leaving such code untested.

But be wary, and always remember that:

•Code coverage does not translate directly into quality. There is no simple relation here. Period.

•"Covered" does not mean it is really "tested".

•Even 100% code coverage does not mean your tests cover all business requirements. They only signify that every statement in your production code gets executed during tests. This is not the same.

•It is completely useless in cases of multithreaded testing. Take, for example, code presented in Section 6.10, which is meant to be thread-safe. Imagine you run only a single test case against such code, using only one thread. This is obviously a nonsensical test, because it does not evaluate the most important feature of this code – no matter that you were able to achieve 100% code coverage thanks to such a useless test.

•Never write a test just to push up code coverage. Each test should aim at covering some important functionality of your code

TDD is probably the best way of achieving high "functional" code coverage. By designing your code with tests, you not only make code coverage reports look nice, but also the coverage values become meaningful. This is because they now reflect the amount of functionalities covered by tests.

244