Automatically check which tests are turned off

When: Failing tests are disabled, not moved to a separate pack

Some teams don’t move failing tests into a separate pack, but they disable them so that a known failing test won’t break the overall validation. The problem with this approach is that it’s easy to forget about such disabled tests.

A separate pack allows us to monitor the problems and ensure that they eventually get ixed or that we don’t waste time troubleshooting a similar problem again until the issue gets ixed. We can’t do this easily with tests that are disabled. An additional problem is that someone might disable a high-priority failure without understanding that it should actually be ixed straightaway.

If you have tests that are disabled, automatically monitor them.

The team at Iowa Student Loan has an automated test that checks to see which tests are disabled. Tim Andersen said:

People were turning tests off because we needed a decision or we were writing a new test and weren’t sure how this old test it in. There were conversations that never got followed up on, or people just forgot to turn the test back on. Sometimes the tests were turned off because people were working on it and there was no code behind it yet.

We used FitNesse to ind the tests that were turned off, and we had a page that checked all of those test names. We’d use that to list the tests that were intentionally turned off and put a JIRA [an issue-tracking system] ticket next to each test. So a turned-off list acts as another test. It has to match what you say you turned off. At the end of an iteration, we follow up on those tests that were turned off. We could say, “This is no longer applicable, let’s just delete the test,” or “Oh, there’s a difference and we didn’t really hear back from the business,” and in these cases we’d have to ix the test.

If you opt for temporarily disabling broken tests instead of moving them out into a separate pack, make sure that you can monitor them easily and prevent people from forgetting about the disabled tests. Otherwise, the living documentation system you build will quickly get outdated. Disabling executable speciications is a quick and dirty temporary ix for a broken test, but it defeats the whole point of continuous validation.