Chapter 11. Test Quality

Check Code Coverage

As we noted in Section 11.3, code coverage can inform us about areas of code that are being undertested. Three minutes should be enough time to run the build script and have a look at the code coverage report. What you are looking to find, in such a short period of time, are white areas of untested code. There is no time to ponder over each line, but it is definitely possible to see that some package has 20% code coverage, while the rest is close to 80%20.

…and if there is no build script which would allow you to generate a code coverage report? Well, then you have one more issue to report to your colleagues.

Conclusions

Three minutes will not allow you to perform a real test code review, but it is enough to uncover some major issues with the test code. If this is all you can have at the moment, then fair enough - it is still much better than nothing.

11.5.2. Things to Look For

Now let us assume that we are under significantly less time pressure, and so have time to really look into the test code. Here is the list of things we should be looking for.

Basically, you should pay attention to the same code features as when code reviewing production code. Are the methods short and focused? Has the code been written at the right level of abstraction? Are there any global variables and magic numbers? And so on… In the subsections below, I will be trying to focus on test-related checks.

Some of the hints below are written from the point of view of a technical team leader, responsible for ensuring the quality of all code. For example, analyzing trends in code coverage reports is probably not something you will be doing on a daily basis. Use common sense to create your own checklist of the issues to look for when code reviewing test code.

Easy to Understand

A good unit test is easy to understand. But its readability can be spoilt by many small issues, which you should look out for.

A good test method has a content-revealing name, which gives information on the particular scenario implemented within it (see Section 9.2). Similarly, variables used in tests should be easy to understand: for example, can you tell which variable is an SUT and which are collaborators? Also, variables used within test code should inform you precisely about what their role is: are they here only to satisfy the API of the method being tested, or are they crucial to triggering some behaviour of the SUT or its collaborators? (see Section 11.6.3).

Are the test methods short and focused? They should test a particular feature of the SUT (see Section 10.1), and nothing more. Look for anything that goes beyond just the simplest actions (arrange/act/ assert).

20Please, do not take these values literally! They are only meant to be illustrative. See Section 11.3 for more information on desired code coverage levels.

251

Chapter 11. Test Quality

Can you find any for loops in the test methods, or instances of reflection being used to set up a test fixture? Both have some legitimate use, but usually cover up deficiencies in production code. Any violation of KISS should attract your attention.

Look for hidden dependencies between tests - they make test code much harder to understand. Depending on the order of execution (which is not guaranteed by JUnit), or relying on data created by another test method, should both arouse your suspicions. Also global variables, reused between many test methods, should be treated as a code smell.

Look for calls to some external APIs. Readability issues are often there, especially if called methods take many parameters. If you can not understand such code at a glance, then there is probably some room for improvement there (see Section 11.6.2).

Are test classes inheriting from some parent class? How many levels of inheritance are there? Inheritance kills readability.

A common issue is the mixing up of different styles within the codebase. Some developers value the arrange/act/assert pattern, some are more in favor of the BDD approach, some like to instantiate test doubles within test methods, while others prefer to rely on set-up methods for this, and so on. A good unit testing suite will be consistent in this respect, and while code reviewing you should also take a look at this. However, this is not something to be fixed easily, as it takes time for the team members to converge and agree on a common style, shared by all developers.

Another thing which impacts negatively on readability is making use of custom solutions, instead of relying on what your own testing framework offers. If you find any non-standard approaches to setting up test fixtures (something different from the use of @BeforeXYZ annotations) or running test methods with different parameters (e.g. running tests in for loops instead of using parameterized tests - see Section 3.6), then by all means kick up a fuss about it.

Similarly, you should take a look at the structure of the test classes. Does each particular part of the test class (e.g. data providers, private utility methods, set-up method, etc.) always appear in the same order? If not, this might be something to fix.

The existence of duplicated code fragments might indicate that the "refactor" phase of the TDD cycle (see Section 4.2) is not being treated seriously. On the other hand, if the duplicated code helps to make the tests more readable, I would leave it alone. This is somewhat different from the situation with production code, where repeated code is almost always a code smell (see Section 11.6.6).

Look at assertions. If a test fails, will you know exactly why? Are the right assertions being used? Are assertion messages clear? (see Section 8.4)

If there is any logic involved (e.g. iteration over the collection returned by the SUT to find out if it contains certain values), then shouldn’t it perhaps be encapsulated within a custom matcher class? (See Section 6.6.)

Test doubles are a common source of readability problems. Are the right ones being used? Are the expectations concerning executed methods clear? Is it clear what is verified and what is only stubbed? It is also common for maintainability issues to arise here. Look for overspecified tests (see Section 10.4.1). Are matchers used properly (see Section 6.7)?

The creation of objects (see Section 9.6) can also be a weak point of tests. It is common to see a lot of copy&paste code in the test fixture setup parts of test classes, or to find many obscure private methods calling one another to set domain objects in certain states. Definitely something to have a look at.

252

Chapter 11. Test Quality

Documented

Well-written unit tests usually do not need documentation (see Section 9.3). However, it sometimes transpires that you come across things which you wish had been documented, but are not. For example, the selection of test cases might not be obvious (e.g. "why is this method validated against Dubai and Sydney timezones?"). Probably there is some business explanation for this, which should be added as a comment (sometimes a link to bug tracker issue is all that is required). If it is not there, then you cannot determine whether the test cases are covering all the important scenarios.

Are All the Important Scenarios Verified?

The most important question to be answered is whether all the important test cases are covered by tests. Often you will find only single executions of tested methods, which is definitely not enough to verify their correctness (see Section 6.1). This is a sign of "happy path" testing, and definitely something to fight against.

Another source of undertested code results from concurrency. If your code is intended to be accessed concurrently, then this is exactly how it should be tested (see Section 6.10). If not, then even 100% code coverage is nothing to be proud of.

We have already discussed one area of code coverage usefulness for test code reviews (that is, its ability to uncover areas not touched by tests at all). However, more can be learned from studying the coverage report, even if the whole idea of measuring test code quality by looking at code coverage is flawed (see the discussion in Section 11.3).

Study the coverage reports to find answers to the following questions (all of them can uncover potential issues):

•Can you see a pattern of untested code across multiple classes? I often find that there are no tests for exceptions thrown. Usually this happens when real collaborators are used instead of test doubles, which makes it hard to simulate some possible paths of code execution.

•If you happen to have historical data on executed tests (your continuous integration server should provide such reports), then see how the number of tests and code coverage measurements change. Ideally the number of tests should grow, and code coverage should at least fluctuate around the same values. If you see some different behaviour, you need to inspect it further.

Additionally, you could apply mutation testing verification (see Section 11.4) to find weak spots in your tests.

Run Them

Some of the issues relating to running tests during code reviews have already been discussed. However, it is now time to have a closer look at them.

Remember, always execute the tests which are under code review.

Take a look at the build script. See if there are any conditions under which tests are not run (e.g. Maven profiles). Make sure the tests are not skipped.

253