Chapter 11. Test Quality

A job worth doing is worth doing well.

— An old saying

Who watches the watchmen?

— Juvenal

Developers are obsessed with quality, and for good reasons. First of all, quality is proof of excellence in code writing - and we all want to be recognized as splendid coders, don’t we? Second, we have all heard about some spectacular catastrophes that occurred because of the poor quality of the software involved. Even if the software you are writing is not intended to put a man on the moon, you still won’t want to disappoint your client with bugs. Third, we all know that just like a boomerang, bad code will one day come right back and hit us so hard that we never forget it - one more reason to write the best code possible!

Since we have agreed that developers tests are important (if not crucial), we should also be concerned about their quality. If we cannot guarantee that the tests we have are really, really good, then all we have is a false sense of security.

The topic of test quality has been implicitly present from the very beginning of this book, because we have cared about each test being readable, focused, concise, well-named, etc. Nevertheless, it is now time to investigate this in detail. In this section we shall try to answer two questions: how to measure test quality and how to improve it.

In the course of our voyage into the realm of test quality, we will discover that in general it is hard to assess the quality of tests, at least when using certain tools. There is no tool that could verify a successful implementation of the "testing behaviour, not methods" rule (see Section 10.1), or that could make sure you "name your tests methods consistently" (see Section 9.2). This is an unhappy conclusion, but we should not give up hope completely. High-quality tests are hard to come by, but not impossible, and if we adhere to the right principles, then maybe achieving them will turn out not to be so very hard after all. Some tools can also help, if used right.

11.1. An Overview

When talking about quality we should always ask "if" before we ask "how". Questions like "would I be better off writing an integration test or a unit test for this?", or "do I really need to cover this scenario – isn’t it perhaps already covered by another test?", should come before any pondering of the design and implementation of test code. A useless and/or redundant test of the highest quality is still, well, useless and/or redundant. Do not waste your time writing perfect tests that should not have been written at all!

In the rest of this chapter we shall assume that a positive answer has been given to this first "if" question. This is often the case with unit tests which, in general, should be written for every piece of code you create.

Test Smells

When dealing with production code, we use the "code smell"1 to refer to various statements in code that do not look (smell) right. Such code smells have been gathered, given names and are widely recognized

1http://en.wikipedia.org/wiki/Code_smell

235

Chapter 11. Test Quality

within the community. Also, there are some tools (e.g. PMD2 or Findbugs3) which are capable of discovering common code smells. For test code there is a similar term - "test smell" - which is much less commonly used. Also, there is no generally agreed on list of test code smells similar to that for production code4. What all of this adds up to is that the problem of various "bad" things in test code has been recognized, but not widely enough to get "standardized" in the sort of way that has happened with production code.

Let me also mention that many of the so-called code or test smells are nothing more than catchy names for some manifestation of a more general rule. Take, for example, the "overriding stubbing" test smell, which occurs if you first describe the stubbing behaviour of one stub in a setUp() method, and then override it in some test methods. This is something to be avoided, as it diminishes readability. The thing is, if you follow a general rule of avoiding "global" objects, you will not write code which emits this bad smell anyway. This is why I am not inclined to ponder every possible existing test smell. I would rather hope that once you are following the best programming practices - both in production code and test code - many test smells will simply never happen in your code. So in this part of the book I will only be describing those bad practices which I meet with most often when working with code.

Refactoring

When talking about code smells, we can not pass over the subject of refactoring. We have already introduced the term and given some examples of refactoring in action, when describing the TDD rhythm. In this chapter we will discuss some refactorings of test code. As with production code refactoring, refactorings of test code help to achieve higher quality of both types: internal and external. Internal, because well-refactored tests are easier to understand and maintain, and external, because by refactoring we are making it more probable that our tests actually test something valuable.

At this point a cautious reader will pause in their reading – feeling that something is not quite right here. True! Previously, I defined refactoring as moving through code "over a safety net of tests", whereas now I am using this term to describe an activity performed in the absence of tests (since we do not have tests for tests, right?). Yes, right! It is actually a misuse of the term "refactoring" to apply it to the activity of changing test code. However, the problem is that it is so commonly used this way that I dare not give it any other name.

Code Quality Influences Tests Quality

Another point worth remembering is that there is a strong relation between the quality of your production code and test code. Having good, clean, maintainable tests is definitely possible. The first step is to write cleaner, better designed and truly loosely-coupled production code. If the production code is messy, then there is a considerable likelihood that your tests will also be. If your production code is fine, then your tests have a chance of being good, too. The second step is to start treating your test code with the same respect as your production code. This means your test code should obey the same rules (KISS, SRP) that you apply to production code: i.e. it should avoid things you do not use in your code (e.g. extensive use of reflection, deep class hierarchies, etc.), and you should care about its quality (e.g. by having it analyzed statically, and put through a code review process).

2http://pmd.sourceforge.net/

3http://findbugs.sourceforge.net/

4At this point I would like to encourage you to read the list of TDD anti-patterns gathered by James Carr - [carr2006]

236

Chapter 11. Test Quality

Always Observe a Failing Test

Before we discuss ways of measuring and improving the quality of tests, let me remind you of one very important piece of advice, which has a lot to do with this subject. It may sound like a joke, but in fact observing a failing test is one way to check its quality: if it fails, then this proves that it really does verify some behaviour. Of course, this is not enough to mean that the test is valuable, but provided that you have not written the test blindfolded, it does mean something. So, make sure never to omit the RED phase of TDD, and always witness a failing test by actually seeing it fail.

Obviously, this advice chiefly makes sense for TDD followers. But sometimes it could also be reasonable to break the production code intentionally to make sure your tests actually test something.

11.2. Static Analysis Tools

Now that we have some basic knowledge regarding tests quality, let us look for the ways to measure it. Our first approach will be to use tools which perform a static code analysis in order to find some deficiencies. Are they helpful for test code? Let us take a look at this.

You will surely use static code analyzers (e.g. PMD or Findbugs) to verify your production code. The aim of these tools is to discover various anti-patterns, and point out possible bugs. They are quite smart when it comes to reporting various code smells. Currently, it is normal engineering practice to have them as a part of a team’s Definition of Done5. Such tools are often integrated with IDEs, and/or are a part of the continuous integration process. A natural next step would be to also use them to verify your test code and reveal its dodgy parts.

And you should use them – after all, why not? It will not cost you much, that is for sure. Simply include them in your build process and - voila! - your test code is verified. But…, but the reality is different. In fact, you should not expect too much when performing static analysis on your test code. The main reason for this is, that your test code is dead simple, right? There are rarely any nested try-catch statements, opened streams, reused variables, deep inheritance hierarchies, multiple returns from methods, violations of hashCode()/equals() contracts, or other things that static code analyzers are so good at dealing with. In fact your tests are this simple: you just set up objects, execute methods and assert. Not much work for such tools.

To give a complete picture I should mention that both PMD and Findbugs offer some rules (checks) designed especially to discover issues within JUnit testing. However, their usefulness is rather limited, because only basic checks are provided. For example, they can verify whether6:

•a test method has at least one assertion,

•two double values are compared with a certain level of precision (and not with exact equality).

5See http://www.scrumalliance.org/articles/105-what-is-definition-of-done-dod and other net resources for more information on Defintion of Done.

6Please consult Findbugs and PMD documentation for more information on this subject.

237