Appendix A. Automated Tests
guaranteed that the code really behaves in the way they describe. This is very different from "traditional documentation". The documentation written in the form of tests is always up-to-date.
A.2. When and Where Should Tests Run?
Running tests is not about making anyone happy: not you, not your manager, not even some book authors. It is all about getting rapid feedback about your work. In particular, it tells you whether your code adheres to the specified requirements (which you have written in the form of tests).
Because rapid feedback is so valuable, an obvious answer to the "when" question is: "as often as possible". This advice can be implemented right away for unit tests, which are ultra-fast by their very nature. But for other types of test you should adjust their execution frequency so they do not slow development down. If your integration tests run for 10 minutes, then it would probably be better if they were not run every time a few new lines of code were added. But why not, say, run them thrice a day – e.g. when you go for dinner, or for coffee? Well, that probably makes sense.
The answer to the "where" question is more complicated, because two factors are in play. The most important thing is to make sure that your application works on the production environment. It is really nice to know that it works on your local box, but customers rarely care about this. What they need, and what you should deliver, is an application that runs on the production environment. So ideally your end- to-end tests should be run right there. It also means that your unit tests and integration tests should be run outside developers’ boxes to avoid the "it worked on my machine" syndrome. The first thing you can do in connection with this is set up a continuous integration server (CI).
All in all, then, to shorten the feedback time you should run every possible test on the developer’s machine, but in order to know the quality of your application from the customer’s point of view you should run them on a production-like environment2.
The solution is rather obvious. Your tests will be run more than once, so you can:
•find bugs as soon as possible,
•eliminate the threat of having code that only works on the developer’s box. You will also need to maintain more than one test execution environment.
If I were to give some pieces of general advice, they would be the following:
•Keep your unit tests separated from other types of test,
•Run unit tests frequently (i.e. before every commit) on developers’ machines,
•Your CI server should run unit tests on every commit, or by frequent polling for SCM changes (e.g. every minute),
•Run integration and/or end-to-end tests on developers’ machines after a bigger chunk of functionality is finished (or run them at least once a day if this is not the case),
•Integration tests should be run regularly on the CI server – at least once a day (i.e. nightly),
2By ‘production-like environment’ I mean an environment that is as close as possible in every respect to a real production environment. Ideally this should be the production environment itself, but that is rarely available for testing.
268
Appendix A. Automated Tests
•End-to-end tests should be run regularly, e.g. once a day, and every time a bigger chunk of functionality is finished3.
Developers’ machines are the first line of defense. The CI is a testing workhorse. The production environment is where the value of your application is proved (or debunked). Utilize all of them for testing, to make sure your software works as required
If you are attracted by the idea of running tests all the time, then take a look at Infinitest4 and JUnitMax5 Eclipse plugins. Both bring "continuous integration to your IDE" by running tests repeatedly in the background as you code.
3End-to-end tests vary a great deal, depending on the application you are writing, so it is hard to generalize about them. 4http://improvingworks.com/products/infinitest
5http://junitmax.com/
269