Appendix C. Test Spy vs. Mock

As we mentioned in Section 5.2, test spies and mocks have a great deal in common. Their purpose is identical: they are there to verify the correctness of calls from the SUT to its DOCs (so-called "indirect outputs"). However, there are some subtle differences between the two, which are definitely worth discussing. As you will soon learn, some of the differences between the two are clear, and some are still being discussed by the community.

(DISCLAIMER) It is really hard to compare such theoretical entities as the mock and the test spy without comparing their specific implementations - like the ones provided by EasyMock and Mockito. I have put a lot of effort into making sure that this section abstracts out from specific and particular solutions, but I think that it (inevitably) reflects the current state of mocking and test-spying frameworks.

C.1. Different Flow - and Who Asserts?

Let us compare two snippets of test code. The first one, shown in Listing C.1, has been written with a test spy (using the Mockito framework). The second snippet, shown in Listing C.2, has been written with a mock (using the EasyMock framework).

Before looking at both tests, let us remind ourselves that a typical test without test doubles (like the ones discussed in Section 3.9) consists of three parts: : first it arranges things (i.e. sets up the SUT and DOCs), then it acts (i.e. calls the SUT’s methods), and finally it asserts (i.e. verifies returned values).

Listing C.1. Sample test spy test

// arrange

SomeClass sut = ...;

ClientDAO testSpyClientDAO = mock(ClientDAO.class); sut.setClientDAO(testSpyClientDAO);

// act sut.deleteClient(req, resp);

// assert verify(testSpyClientDAO).deleteClient("1234");

The SUT gets created (it does not matter how).

Assertions are done explicitly by test method.

In the case of the test spy the three phases are clear. However, in Listing C.2 we can see that this is not the case with the mock. The assertion phase happens in two places within the code - when the expectations are set, and when the framework is asked to actually verify whether they have been met.

278

Appendix C. Test Spy vs. Mock

Listing C.2. Sample mock test

// arrange

SomeClass sut = ...;

ClientDAO mockClientDAO = createMock(ClientDAO.class); sut.setClientDAO(mockClientDAO);

//assert - part I expect(mockClientDAO.deleteClient("1234")); replay(mockClientDAO);

//act

sut.deleteClient(req, resp);

// assert - part II verify(mockClientDAO);

The SUT gets created (it does not matter how).

Creation of mock, but this time using the static createMock() method of EasyMock.

No assertions are done directly by the test method. Instead, some expectations are expressed regarding the mock’s behaviour. Real assertions are done within the mock object after the test method has called the verify() method.

Informing the mock that the expectation part is over, and that now the real execution begins.

As the Listing C.1 and Listing C.2 show, there is a difference when it comes to the ordering of actions in the tests. Tests based on test spies follow the "arrange, act, assert" pattern. However, tests based on mock objects reverse the order by first stating our expectations, and then exercising the code. This feature of mock-based tests may come across as unintuitive, and is one of the main reasons for switching to a test-spy approach.

C.2. Stop with the First Error

The difference in the order of actions described in the previous section has some impact on the way tests fail. The difference is as follows:

•mock-based tests fail during the execution of methods, as soon as any of them do not match expectations,

•test-spy-based tests fail during the verification phase, after all methods have been executed.

This has some impact on the ability of mocks and test spies to provide a good, descriptive error message. It seems that test spies are able to provide more detailed information in assertion messages than mocks (based on precisely the kind of information about method calls which would never arise, and thus would not be available, in the case of quickly failing mocks). On the other hand, mocks fail immediately where some runtime information is available. If our test spy is not recording this information, it will not be available when it fails.

C.3. Stubbing

Some authors differ in their opinions as to whether test spies also have stubbing capabilities. However, the majority, including Gerard Meszaros, the father of test doubles taxonomy, answer this question in the affirmative. Mockito also provides stubbing capabilities for its test spies.

279

Appendix C. Test Spy vs. Mock

C.4. Forgiveness

There is a general idea that test spies are more "forgiving" than mocks. This means that test spies let us specify which interactions are to be verified, and turn a blind eye to the rest. In fact, this is one of the notable features of the Mockito test-spying framework. Its author, Szczepan Faber, promotes such an approach as a way of testing only what should be tested at some given point in time, without paying attention to interactions that are beyond the scope of that particular test1. The motivation behind such behaviour is to avoid overspecified tests, which are a big problem to maintain (see Chapter 10, Maintainable Tests). However, the idea is disputed within the community, and some people would rather vote for stricter verification of the SUT’s actions.

Without getting involved in the details of this debate, let us say that in general mock frameworks are stricter than Mockito. However, some of them – e.g. EasyMock - support both styles of testing. By default, its mocks are "strict" (which means they fail tests if any unexpected interaction occurs), but also allow one to use so-called "nice" mocks, which verify only selected interactions - exactly the way the test spies of Mockito do.

To conclude, "forgiveness" is not supported by test spies exclusively, even though they do seem more inclined towards it than mock objects.

There is some unclarity about how the terms "nice" and "strict" are to be understood when used to characterize mocks. By "strict" mocks, [meszaros2007] has in mind mocks which will fail if the received method calls are in a different order than the expected one. Mocks which are "nice" or "lenient" will accept a different order, and only pay attention to the methods and values of parameters. But based on my observations I would say that most other people follow a different definition of the meaning of the terms "nice" and "strict" when used here - exactly like that given previously in this section.

C.5. Different Threads or Containers

If the SUT is running inside a different "environment" than the test code which invokes it (e.g. some kind of container, or a different thread), then it may happen that this environment will eat up all the exceptions and messages reported by a mock in the runtime. In the case of a test spy, which acts like a callback object, all errors will be reported after the execution has finished, from within the same environment (e.g. thread) which the test code uses. That way no information gets lost.

C.6. Conclusions

Historically, we all used mocks. They were mentioned in the title of the famous "Mock Roles, Not Object" paper ([freeman2004]), which brought mocks to the masses, and for a long time no distinctions between test doubles were made at all - we just called all of them "mocks". Today, things are different, because we have learned from experience that there are, indeed, differences between test doubles. They also exist between mocks and test spies - even though both can be used (almost) interchangeably.

There are some reasons why I would consider it preferable to use test spies rather than mocks. Of these, the two most important are that they allow me to keep to the arrange/act/assert pattern, and that they

1Please refer to http://monkeyisland.pl/2008/07/12/should-i-worry-about-the-unexpected/.

280

Appendix C. Test Spy vs. Mock

promote the writing of focused tests, which are only interested in selected aspects of the code I intend to test.

281