Chapter 8. Getting Feedback

Now the error message will be much more informative:

Listing 8.7. Fixed assertion error message

java.lang.AssertionError:

Expected :Client{name='Mary Hopkins'} Actual :Client{name='John Brown'}

8.4.3. Use the Right Assertion Method

And while we are discussing the informativeness of failure messages, please consider the following comparison of different assertions.

Table 8.1. Comparison of assertion failure messages

Table 8.1 shows clearly how the misuse of assertTrue() leads to a weak assertion message, leaving the developer with no information as to the cause.

8.5. Logging in Tests

The Loudmouth: A unit test (or test suite) that clutters up the console with diagnostic messages, logging messages, and other miscellaneous chatter, even when tests are passing. Sometimes during test creation there was a desire to manually see output, but even though it’s no longer needed, it was left behind.

— James Carr 2006

Let’s make it absolutely clear: you do not need logs for unit tests. They are only there because we are so used to the fact that we do verify our software by reading through billions of logs. This is unnecessary with unit tests. It may even lead to some problems, especially if you run your tests from the command line.

This section is aimed mainly at developers who use the command line for running tests. IDEs are quite good at only presenting users with the results of tests, so even thousands of log lines will not be a problem for them.

Watching screens showing logs rolling for a few seconds at the speed of light will not give you anything. You cannot read it, and if the test fails, it should give you exact information on what happened, right? I understand that for integration tests it might be worthwhile to watch Spring or JBoss logs getting printed and see there are no warnings there, but what is the point for unit tests?

185

Chapter 8. Getting Feedback

And the worst thing of all is when you test for expected exceptions and they are printed within the logs which roll over your screen at such a rapid pace. The first time you see it, you shiver, and check it immediately, and then you sigh with relief. And later, after n-th execution of the tests, you get used to exception stack traces being printed, and you start ignoring them. And then comes a day, when some other exception happens, and you do not react at all. You simply do not care, because "these exceptions were always there", and you do not try to investigate it. In fact, you are so used to the idea of a build’s being successful even if some exceptions have been logged, that you do not bother to get to see the final result of a build. You assume it passes, when in reality it has just failed. And then you commit the code and your information radiator goes red, and your team starts yelling at you. Uh, no good…

Personally, I am perfectly happy coding with my logs turned off (by having a separate configuration of logging frameworks) and a custom test listener which prints single letters for each test - . (a dot) if it has passed, F if it has failed, and S if the test has been skipped (exactly as described previously in Section 8.3). In the event of test failures, I rely solely on the assertion messages.

Even if you do not want to switch your logs off, the thing to remember is that you should stop relying on your logs. The tests should be automated – they should be precise in pointing out the problem, which means that no logs-hunting should be necessary to understand the cause of failure.

8.6. Debugging Tests

Debugging? Haven’t we already mentioned that by testing we can save hours spent on debugging sessions? Well, yes. Still, sometimes you run your tests and you do not understand why they are not working. You stare at the screen and cannot figure out what is going on. Do not stare any longer. Fire up a debugger and look inside.

With unit tests you get direct support from your IDE. Debugging test code does not differ from normal debugging sessions. If you are already familiar with debugging, there will not be much that is new to learn. The same rules apply as when debugging a running application. This should not be a surprise, because tests run your application (even though they do it in some specific, strictly controlled way).

Appendix B, Running Unit Tests covers running a debugging session with Eclipse and IntelliJ IDEA.

8.7. Notifying The Team

A common request is that you notify the team (or some superiors) about the results of your tests. The reason is quite sensible: there are some people who ought to know whether all the tests have passed or not. Perhaps because they will be able to fix it, or maybe because they need to know the status of the project. No matter what the reason is, such functionality can be implemented at the level of JUnit (or almost any other testing framework). For example, one could write a custom rule (see Section 6.8.2) which, after all tests had finished, would send an email notification using the Java Mail API3. Even so, I myself would argue that this is not the right approach.

There are two main reasons for saying this. First, you should let your testing framework take care of executing tests and creating reports. This is what testing frameworks are for. JUnit is quite extensible, which means you can enhance it with almost any functionality you wish, but this does not mean you should do so.

3See http://www.oracle.com/technetwork/java/javamail/index.html

186

Chapter 8. Getting Feedback

The second reason is that there are already solutions for the notifications sending issue. I would hope that your team makes use of a continuous integration server, which runs all tests continually. Any popular CI solution will offer a notifications service, and usually much more besides. For example, in the case of Jenkins4, a very popular open-source CI server, there are more than 30 notification plugins5 which allow the sending of information about build results (including test results) by email, Jabber, Skype, Twitter and many more.

To conclude, first you should "use the right tool for the job", and second, in the presence of such a wealth of freely available options, why bother with implementing your own solution?

8.8. Conclusions

In this section we have concentrated on ways of getting feedback information about the execution of tests. First, we covered feedback generated by the IDE. Then we moved to the output generated by JUnit. We learned about the default for this, and explored options for customization. We also did some coding and created our own listeners. In so doing we made JUnit print information that was of value to us, both during the execution of the tests and after they had finished. As the few straightforward examples given here demonstrate, JUnit will furnish us with comprehensive information about the tests that have been executed, and it is up to us how we use it.

To gather even more data on the execution of tests, we also plunged into the world of assertion messages, logging and debugging.

Well, quite a lot of information. Now that you know all of this, how do you plan to use it? Will you rely on the default information returned by your IDE and testing framework, or do you perhaps feel that some customization is required? Remember, you will be running unit tests frequently and checking their results multiple times, so make sure you feel comfortable with the feedback you are getting.

4See http://jenkins-ci.org

5See https://wiki.jenkins-ci.org/display/JENKINS/Plugins

187