Chapter 7. Points of Controversy

Discussion is an exchange of knowledge; an argument an exchange of ignorance.

— Robert Quillen

There are many topics related to writing tests that are still up for debate within the developer community. In this section we will discuss some of these. Sometimes a definite answer will be given, sometimes a variety of options will be presented – for you, dear reader, to decide about.

7.1. Access Modifiers

In the code shown throughout the book, I follow unwritten standards of the Java world relating to the use of access modifiers. In particular, I used the private keyword for almost every instance variable. There is nothing wrong with this approach; however, some people may find it unnecessary.

The point here is that while with production code we tend to be obsessed with encapsulation and information hiding, this is not the case with test classes. They are almost never used by other classes 1, which means that the fact that some field or method is marked with private does not really make code any better. Hence, if you are in the habit of marking everything with the private keyword, then by all means go ahead and adopt this approach with test code as well. On the other hand, if you think that by omitting private you are going to gain some readability in respect of your test code, then do get rid of it. (And don’t let any zealots intimidate you into doing otherwise!)

7.2. Random Values in Tests

At first, testing and random values seem to fit together nicely. Some people apparently think that running tests each time with different values can better prove the robustness of their code than running it with some selected values. In my experience, this is a fallacy, and using random values in tests causes more harm than good. In fact, I can recall instances of randomness being used in tests where it proved futile, but none where it turned out to be worthwhile! This section shows some typical cases that are to be avoided.

There definitely are some areas where random tests are not only valid, but also vital! For example the test suites of Apache Lucene2 contain a lot of randomly generated test cases which allowed them to report an important Java 7 bug3.

Before we start discussing code, however, let me just say that there are many ways to generate random values, including:

•custom "utility methods",

•using libraries for general use; e.g. Apache Commons Lang library4 provides a RandomStringUtils class, which contains a plethora of methods for generating random Strings of the requested length and consisting of different character sets (e.g. only letters, only numbers, a combination of both, etc.),

1This is not 100% true, as sometimes we use inheritance, so some test classes inherit the fields and methods of a base class. 2http://lucene.apache.org

3See http://blog.thetaphi.de/2011/07/real-story-behind-java-7-ga-bugs.html 4http://commons.apache.org/lang/

146

Chapter 7. Points of Controversy

• using specialized test-focused frameworks like Quickcheck5.

The issues that I would like to discuss here are not really related to any particular way of generating random values, but rather to the idea of using randomness in your tests.

7.2.1. Random Object Properties

Let us assume that an SUT (of the UserToPersonConverter class) can translate an object of the User class into objects of the Person class by turning the user’s name and surname into a person’s nickname. The implementation of the SUT is shown below:

Listing 7.1. The UserToPersonConverter class

public class UserToPersonConverter {

public static Person convert(User user) {

return new Person(user.getName() + " " + user.getSurname());

}

}

A test of this class can take advantage of random values. Such an attempt is shown in Listing 7.2.

Listing 7.2. Test of the UserToPersonConverter class

public class UserToPersonConverterTest {

@Test

public void shouldConvertUserNamesIntoPersonNick() { String name = RandomStringUtils.randomAlphabetic(8);

String surname = RandomStringUtils.randomAlphabetic(12); User user = new User(name, surname);

Person person = UserToPersonConverter.convert(user);

assertEquals(name + " " + surname, person.getNick());

}

}

As far as I understand the idea behind the implementation shown in Listing 7.2, random values are to be used because they (supposedly) highlight the robustness of a given test. It is as if the test were to shout "Hey look, every time I execute the test it uses a different user’s name and still passes! Amazing, isn’t it?".

7.2.2. Generating Multiple Test Cases

A natural next step in the direction of making the test even more impressive would be to test the same functionality with more random values, within a single execution of a test method. This can be achieved using data providers, for example:

5http://java.net/projects/quickcheck

147

Chapter 7. Points of Controversy

Listing 7.3. Creation of multiple random test cases

@RunWith(JUnitParamsRunner.class)

public class UserToPersonConverterDataProvidersTest {

private static Object[] getRandomNames() { Object[] values = new Object[100];

for (int i = 0; i < values.length; i++) {

values[i] = $(RandomStringUtils.randomAlphabetic(8), RandomStringUtils.randomAlphabetic(12));

}

return values;

}

@Test

@Parameters(method = "getRandomNames")

public void shouldConvertUserNamesIntoPersonNick( String name, String surname) {

User user = new User(name, surname);

Person person = UserToPersonConverter.convert(user); assertEquals(name + " " + surname, person.getNick());

}

}

This test method is executed 100 times with different random values.

Even if the test in Listing 7.3 looks much more serious than the previous one, it is not really much stronger. It only tricks us into thinking that UserToPersonConverter has now been thoroughly tested. Unfortunately it hasn’t been.

Let us take another look at the implementation of the UserToPersonConverter class (shown in Listing 7.1). Has it been tested more effectively, just because we have passed 100 nonsensical, unpronounceable names (each time of the same length, namely 8 and 12)? I do not think so. The probability that tests 2 to 100 will reveal some bugs not discovered by the first test is minimal. The diversity of test parameters is very limited, and so is the value added by each of the tests. It would not increase, even if we were to up the number of randomly generated values from 100 to 1000.

When it comes to testing, I would rather prioritize the quality of test cases over the quantity. A good quality test case can expose some holes within the production code. I suggest thinking carefully about the possible scenarios (as discussed in Section 6.1) and then making a deliberate choice of some values. My test of the UserToPersonConverter class would definitely contain cases of names and surnames of varied length (including empty strings), with varied capitalization. I might even use some randomness to avoid putting so many names directly in the test code; however, I would make sure that some borderline cases (e.g. empty or extremely long strings) are verified. The variety of my test parameters would definitely be far greater than that generated by the data provider in Listing 7.3.

7.2.3. Conclusions

As you will have observed for yourself, generating multiple tests with random values does not automatically make the test suite stronger. Instead, it can complicate your tests and give rise to a number of issues. However, if randomness is really appropriate within the context of your testing domain, then at least try to do it right. Table 7.1 offers some hints and discusses issues relating to the use of random values in your tests.

148

Chapter 7. Points of Controversy

Table 7.1. Issues with the random values of parameters

aIf you are interested in details, please refer to Jaroslav Tulach’s article at http://wiki.apidesign.org/wiki/RandomizedTest

Some specialized tools, e.g. Quickcheck, provide solutions for some of the issues described above.

I have to admit I am not a big fan of using random values in tests. I do not see many cases where using them would improve the test. In general, I would rather suggest:

•spending more time pondering over important values which should be verified (see the discussion in Section 6.1) instead of generating thousands of random cases.

•using meaningful names instead of random garbage strings.

149