Appendix A. Automated Tests

[…] if it’s common enough to be worth automating, I automate it. If not, I just slog through it.

— Bobby Tables stackexchange.com (2010)

For many years automated tests have been recognized as a great way to make your software robust and bug-free. The idea is so pervasive that there are many tools (build tools, IDEs), and even languages, that facilitate the process of executing such tests. Nevertheless, I have seen enough under-tested projects to be sure that the issue has yet to be taken on board as widely as it should be.

A.1. Wasting Your Time by not Writing Tests

Even if you don’t write tests, you surely perform some other operations to verify that your code works properly. In order to make sure that the code does what you expect it to do, and to find bugs, you can:

•have debugging sessions (with the help of your great IDE),

•add lots of log messages so you can later browse log files,

•click through the user interface of your application,

•preform frequent code reviews.

All of the above techniques have their legitimate uses. Visual inspection is useful. A debugger and logs can sometimes save your life. Clicking through GUI will help you get to know how it feels to be your user. Code reviews will allow you to find various weaknesses in your code. But if these techniques are the only ways in which you verify the proper functioning of your code, then you are doing it wrongly.

The main problem is that all these actions are very time-consuming. Even if your experience allows you to use a debugger very effectively, or you have strong Linux shell skills, making finding stuff in amongst tons of logs a trivial matter, it still takes some time. Clicking GUI cannot really be accelerated – you must wait for the application to fetch the data, for the browser to render it, and for your brain to locate the right place to click. A decent code review cannot be done in just a couple of minutes. Moreover, time required for testing multiplies, if you are to test your application against many software versions or environments (e.g. multiple web browsers, databases or operating systems).

The time factor is crucial: there is, quite simply, no time to repeat the process. You will have to check once that it works and voila! – finished. After you go back to this code (to add/update functionality) you will want to skip the already tested part, because it simply hurts too much to even think of browsing the logs all over again!

Remember, your time and skills are too precious to be wasted on simple, repeatable tasks that can be done more effectively by machines. What you should concentrate on is teaching the machines to do those tasks!

265

Appendix A. Automated Tests

The second issue is that you are trusting your senses, your judgment and your honesty here. This brings with it a few problems:

•you may overlook something (e.g. omit a line of log files, forget to check everything that should be checked, etc.),

•you may misconstrue or forget criteria of verification, accepting failed tests as passed,

•you may fall victim to the "it is not a bug, it is a feature" fallacy when testing, and persuade yourself that it really works fine,

•you may deceive yourself into thinking it works, even if it doesn’t.

Yes, I know, you do not make mistakes, so you cannot possibly miss a single log line, and of course you are 100% honest with yourself… Well, but if you do not ever make mistakes, how come the bug you are looking for happened in the first place? And about honesty: I can only speak for myself, but it does happen, sometimes, that I only see what I wanted to see, and ignore anything that speaks against how I would wish things to be. Has it really never happened to you? No? Are you sure?

What makes it even more painful is that clicking GUI again and again, or browsing log files for the n- th time, is so boring! Your mind will scream "get me out of here, I want to do some coding!", and it will likely tell you "everything works fine", just so you move on to some more interesting tasks.

In short, verification methods that are not automated suffer from the following disadvantages:

•they are time-consuming and, as such, the first things to be abandoned when a deadline nears,

•verification criteria might not be clear, and the outcome of the verification process can be distorted by human error,

•they are boring, so people do them without due care and attention, or avoid doing them at all,

•they can be hard to repeat exactly (e.g. it is easy to omit some steps during the configuration or execution phases of the test),

•it may be hard to deduce from log files where the source of a bug is, and sometimes lengthy investigation is required to find it,

•they are not included in the build process, and are only run long after the new features or changes were actually introduced into the software, making the feedback they give less valuable (i.e. it costs much more to repair damaged parts not discovered right after they have been damaged – see Figure 1.5),

•people might easily disdain what they see (e.g. "this exception has nothing to do with the functionality being tested; it was always there, and it is harmless"), it is quite easy to just sweep issues under the rug (…no one else has noticed this exception in the logs, have they?).

In my experience, the majority of errors occur when code gets changed, not when it is written for the first time. When you implement a new functionality the list of requirements is usually clear. You simply implement them one by one, making sure that everything works fine. This is simple. More problems emerge when you are asked to introduce some changes. Ooops, the original list of requirements is long gone, the people who wrote the original code are not around anymore, and your manager assumes that

"adding this small piece of functionality shouldn’t take long, should it?" And then, if you don’t have

266

Appendix A. Automated Tests

a safety net of tests, you are in trouble. While adding or changing new functionality you are likely to break something that used to work fine. Such mistakes result in so-called regression bugs. When tests are automated it makes it harder for them to creep in.

A.1.1. And what about Human Testers?

Are human testers still required? Is there a place for them in the development cycle? The short answer to this question is: "it depends". It depends on the type of application you are working on – on the technology and the testability of your software.

It is outside the scope of this book to discuss this (fascinating) topic. However, since I am putting a lot of effort into trying to persuade you to automate your tests (which will, in effect, diminish the role and responsibility of human testers), I have to tackle this subject, if only very briefly.

Looking at the current tendencies in the field of development, I see a major change happening in the way development teams are being organized, especially with respect to the role of testers. A new role – of "developer in tests" – has been created, and its name stresses the importance of developers’ tests as a vital part of every development. Code and tests are being developed in parallel, and every piece of code committed to the SCM1 is tested (on different levels – from unit tests to end-to-end tests). Automation, and continuous integration, is our daily bread right now. What is more, some teams have achieved an even higher level of confidence in their automated tests, which allows them to continuously deploy automatically tested software right up to the production stage.

There is still a place for human testers. However, the situation is changing before our very own eyes, and the role and place of testers is being redefined. It is expected that you – a developer, I suppose – will also have some testing skills.

A.1.2. One More Benefit: A Documentation that is Always Up-To-Date

[…] ordinary narrative documentation is more susceptible to drifting from the implementation of the program and will thus become outdated (e.g., design changes, feature creep, relaxed practices in keeping documents up-to-date).

— Wikipedia on Unit Testing

Documentation lies. It is rarely up-to-date, and reading it often leaves many questions unanswered. It makes no difference whether it is a separate document or Javadocs comments. In general, writing useful documentation and maintaining it throughout the lifetime of a project is a very hard task. A task that we repeatedly fail to perform successfully.

I consider tests to be part of documentation. Not the whole of the documentation, but a vital part of it. Tests will not tell you about the business purpose of an element, neither will they reveal why some code fragment looks the way it does. They have to be accompanied by other documentation that fills in these gaps. But they will leave no room for doubt about how your code should be used, and what happens when you execute method x with parameters a and b. And if you run them frequently, then you are

1SCM, Source Code Management, relates to all kind of software tools which allows one to track file versions. Commonly used tools in software development are Subversion, CVS or Git. For more information see http://en.wikipedia.org/wiki/ Source_Code_Management.

267