166 Speciication by Example

Set up a dedicated continuous validation environment

Continuous validation has to work in a reproducible way to be reliable. In some larger organizations it’s harder to get a new set of machines than it is to hire a known poisoner as a chef in the company cafeteria, but ighting for better equipment is well worth it. Many teams tried to use the same environment for demonstrating features to business users, manual testing, and continuous validation. This regularly caused data consistency issues.

Without a dedicated environment, it’s hard to know whether a test failed because there’s a bug, whether someone changed something on the test environment, or whether the system is just unstable. A dedicated environment eliminates unplanned changes and mitigates the risk of unstable environments.

Employ fully automated deployment

Once we have a dedicated environment, we want to ensure that the software is deployed in a reproducible way. Unreliable deployment is the second most common cause of test result instability. For many legacy systems, deployment is a process done overnight that involves several people, lots of coffee, and ideally a magic wand. When we have to deploy once every year, this is acceptable. When we have to deploy every two weeks, it becomes a major headache. For continuous validation, we might need to deploy several times a day, and magic-aided manual deployment is completely unacceptable.

Without a fully automated deployment that can reliably upgrade a system, we’ll frequently get into situations where many tests suddenly start failing and someone has to spend hours troubleshooting to ind the culprit, only to hear “but it works on my machine” from the back of the room.

This eliminates the dependency of executable speciications on a particular environment and makes continuous validation much more reliable. It also makes problems easier to troubleshoot, because developers can use any environment to reproduce problems.