Mocks: What and why? A directed ramble

Someone recently asked me about mocks.  As I hadn’t yet put my thoughts in writing, I do so here.  Hopefully there’s some helpfulness scattered here somewhere…

“I guess we can only integration test this code – it pulls in the world”

As you start to write unit tests for your code, you quickly find that while there are a few methods that are easy to test, you quickly get bogged down with collaborators.  “Oh, I guess I have to set up a real database server with test data out there somewhere or this method can’t be tested…” or “Rats, I don’t want to have my test run the part that initiates a transaction with this remote internet service, because it will result in charges to the company”, or “These blocks of code only run if the collaborator throws some obscure error, and I don’t have any way of reliably creating ”

More generally, here are some of the challenges to dealing only with integration tests*:

• Setting up the infrastructure for the test takes time and effort – sometimes involving configuring and maintaining database servers or setting up accounts with third parties
• Tests tend to run slowly because they must talk to the database, place a phone call, really perform the complex transaction processing, etc.
• When a test breaks, additional time and effort is needed to isolate the cause, as there are many moving parts
• Due to the infrastructural setup required, the test itself is more complex and prone to bugs.  Time must be spent diagnosing in which class the problem lies (potentially through many layers of the system, sometimes spanning servers, DLLs, and codebases)

Dealing gracefully with collaborators while unit testing is what mocks are all about.

*(See also the earlier article, Why not just integration tests?)

Dealing with the Collaborators

As the EasyMock page says:

Unit testing is the testing of software units in isolation. However, most units do not work alone, but they collaborate with other units. To test a unit in isolation, we have to simulate the collaborators in the test.

A Mock Object is a test-oriented replacement for a collaborator. It is configured to simulate the object that it replaces in a simple way. In contrast to a stub, a Mock Object also verifies whether it is used as expected.

Examples

The syntax varies with the mock framework: a couple of examples from the HippoMocks 3.0 tutorial follow:

    mocks.ExpectCall(barMock, IBar::c).With("hello").Return(42);

and (here’s a big reason I can’t ignore mocks):

    mocks.ExpectCall(barMock, IBar::c).With("hello").Throw(std::exception());

See that? Expect the c() method to be called and throw the specified exception.  This can be really valuable in testing your error handling – you don’t have to laboriously set up a complex net of conditions to get a subsystem to really throw an exception – you just tell the mock object to throw it and that’s that.

Mocks not a replacement for integration tests

Mocks aren’t a total replacement for integration tests.  Many times the problem is not in class A nor class B, but in their collaboration.  Integration tests are valuable to prove that the collaboration works as expected.  But it is faster and easier to diagnose a problem in the collaboration of units that have been thoroughly unit tested in isolation.  To put it the opposite way, when you’re working with a group of collaborating units that haven’t been thoroughly tested in isolation, “the problem could be anywhere”!

Limitations and downsides of mocking

It’s a Simulation

When using mocks to test your unit in isolation, you’re simulating interactions with the unit’s collaborators.  A risk inherent in simulation is the risk that the simulated interaction diverges from interactions that use the real collaborator.

Coupling between the test and the collaborators’ implementation

In insulating a unit from its collaborators, mocks couple your tests to those collaborators.  When you change the way calls to a collaborator work, you have to adjust mocks’ expectations  as well.  This coupling can decrease the flexibility of your code if not understood and managed  (this answer to “When should I mock?“‘s Mock shortcomings section gives examples of this problem).

I’ve experienced this kind of brittleness, and it is an important consideration.  My hope is that such brittleness can be avoided by relaxing the expectations down to the minimum needed (see for instance the “2. More Complex Stuff” section of the HippoMocks 3.0 Tutorial ).  It’s a challenge to do this right the first time, though, because the brittleness of a test using mocks may not be apparent to you until time passes and the collaborator changes its implementation, breaking a test.  It may be a learning process for mock users, to develop a standard practice for the right kind of looseness that supports  .

Why Not Just Fakes, or Ad-Hoc Mocks?

There are ways to get independent unit testability without using a mocking framework.

• You can create your own fakes (i.e., create a testing implemention of your collaborator’s interface) — certainly handy at times
• You can create your own mocks without using a framework (the article Should we use a mocking framework?)

The thing to watch about these approaches is the amount of boilerplate code that’s required.   During my experience in Java-land, using EasyMocks in conjunction with Unitils reduced the boilerplate code down to a really small amount.  It was great.

Which Framework?

I’ve written up some initial results on that front as well.

Conclusion

While using mocks without care can lead to brittle tests, their power for easily testing the unit under test’s responses to interactions with a collaborator cannot be ignored.  I hypothesize that by setting up the most relaxed expectations that fulfill the test’s need, the brittleness problem can be avoided; but more study (or at least more Googling) needs to be done to prove or disprove this.  In the meantime, I believe the right course of action is to proceed (with our eyes open) with the use of mocks, since the current alternatives, including having integration tests but no unit tests, must leave us in the position of leaving certain code untested.  I don’t see how it wouldn’t anyway…

Shoot, now I need to find a good C++ code coverage tool, to prove it!  : )

Mock framework for C++: an introductory flounder

In a prior project in Java, we used EasyMock as our mocking framework.  There were situations where being able to mock things really opened up the testing possibilities.  So today I looked around for what kind of mocking frameworks are available for C++.

On stackoverflow.com, someone asked about good mock frameworks for C++.  Four are listed:

AMOP
Googlemock (the video calls it gMock, but gMock seems to be for Groovy, not C++…)
HippoMocks
MockPP

The HippoMocks page points to these and some others:

MockitoPP
MockItNow
M0cxx0r
Moxy

First Impressions and Reasoning

I was first drawn to Googlemock because it might be well-implemented and better supported, being from Google.

But, it’s based on some kind of ugly looking macros.

AMOP doesn’t have the ugly looking macros.

But, it may have ugly-looking member function pointer syntax instead.

HippoMocks may have better syntax than either of the other two (need to check this).

And I appreciate how the author provides the links to other available mocking frameworks and a comparison between HippoMocks and the other frameworks, and the answers he gave on the Boost mailing list.

That’s all I know so far (and a little more!)

The phantom mock

A couple of days ago, a co-worker came by with an interesting problem.  His unit test class was failing, with a mock saying that it got more calls to a certain method than it had been set up to expect.

The odd thing was, the test class we were looking at didn’t declare any mocks!

A Clue

The failing test class passed when run by itself, failing only when run with the rest of the module’s unit tests…

The Problem

The problem turned out to be that a test class that ran earlier in the test run was injecting a mock into a singleton and not cleaning up after itself.  The singleton was later used by the failing test, which unwittingly made use of the mocked guts of the singleton, resulting in the unmet expectations.

A Solution

The offending test just needed to reset the guts of the singleton back to its normal, non-mocked value in its @After or @AfterClass method.

Musings

It seems to me it would be even better to implement the singleton using a factory, to avoid such side-effects popping up in the future (see StartupSvcFactory: why go to the bother? for some discussion about factories and singletons).  True, there’s the extra conceptual overhead of one more (factory) class and two more interfaces — but I think the resulting cleanness of the test code makes it worth it, at least if you ever need to mock what the singleton provides (as we did in this example).

StartupSvcFactory: why go to the bother?

In a previous post I had said we had

…changed our can’t-mock-dis Startup service from a plain ol’ Singleton…to an Abstract Factory so that…we can now mock out the Startup service and its application-context-creatin’ slowness when we want to.

But some muddled thinking had snuck in on my part: New Ben notes that you don’t need a factory to mock an interface.  Why didn’t we just make the StartupSvc class an interface and mock that interface?  Why go to the bother of creating a StartupSvcFactory as well?

Here’s the proof:

1. Assume that a factory is not needed
2. StartupSvc is a class, and getStartupSvc() is a static method on it
3. You can’t have a static method on an interface (a rule of the Java language), and a factory is not needed (by assumption, #1), so we’ll make getStartupSvc() an instance method of StartupSvc.
4. To call getStartupSvc() as an instance method, you must already have an instance of a StartupSvc on which to call it, BUT because StartupSvc is a singleton, the only way to get an instance in the first place is by calling getStartupSvc().  When getStartupSvc() as an instance method, there is no way to call it.

∴ a factory is needed.

The factory then takes care of managing the singleton-ness of the StartupSvc and holds the static StartupSvc reference.

How does this help mockability?  Client code then holds a reference to a StartupSvcFactory (interface) and is injected with a StartupSvcFactoryImpl whose getStartupSvc() instance method returns a real StartupSvcImpl — but unit tests inject a mock StartupSvcFactory and can then simulate different behaviors coming from the StartupSvc that are difficult to reproduce, or just slower, with the real thing.

Unitils and the class that had no default constructor

Unitils is so helpful for unit testing.  The top portion of my unit tests tend to look something like this:

@RunWith(UnitilsJUnit4TestClassRunner.class)
public class BongoTest {
    @Mock
    @InjectIntoByType
    private Foo foo;

    @Mock
    @InjectIntoByType
    private Baz baz;

    @TestedObject
    private Bongo bongo;

    //...

And then Unitils automatically instantiates a Bongo and injects the Foo and Baz mocks into the Bongo’s fields of those respective types.  My @Test methods can just start testing Bongo’s methods without worrying about the mock creation and injection and tested object instantiation.  I’m free to get right to testing Bongo out.  Nice!

This time, though, there was a twist: my TestedObject (call it a Congo) didn’t have a default constructor — instead, it needed a Foo and a Baz passed to its constructor.  How to do that with the annotations?

On the Unitils cookbook page, it says:

The field annotated with @TestedObject is automatically instantiated with an instance of the declared type, if necessary (i.e. if not instantiated during test setup) and possible (i.e. if the declared type is a non-abstract class and offers an empty constructor).

So it looks like I have to take on a little responsibility myself this time.  It ends up looking like this (notice the new @Before method):

@RunWith(UnitilsJUnit4TestClassRunner.class)
public class CongoTest {
    @Mock
    private Foo foo;

    @Mock
    private Baz baz;

    @TestedObject
    private Congo congo;

    @Before
    public void setUp() {
        congo = new Congo(foo, baz);
    }
    //...

(Everything else is as, er… Before.)

That’s not too bad!

Running with the…

(For some reason, “Running with the bulls” sounds a lot more poetic than “Running with the right JUnit 4 test runner.”)

I was getting a NullPointerException when I started writing the unit test for my little TransactionManager wrapper class.  I am using Unitils to manage all the dependency injection, so my test class looks something like this:

public class TransactionManagerTest {
    @Mock
    @InjectIntoByType
    private PlatformTransactionManager platformTransactionManager;

    @TestedObject
    private TransactionManager transactionManager;

    @Test
    public void testCommitException() {
        transactionManager.commit();
    }

I first got the NullPointerException while trying to test TransactionManager’s getTransaction() method, but it has some more complex logic, so I decided to back off from that and test the commit() method first.

I had expected that when I called transactionManager.commit(), I would get the unchecked exception that commit() throws when you haven’t called getTransaction() first.  But instead, this NullPointerException.

Before looking at the answer, can you guess why that would be?

Oh, wait, I already gave it away at the top of this post. Rats!

(Maybe there’s a reason I’m a programmer instead of a game show host. ;)

Yup, that’s right, folks!  All it was, was, I needed to…

@RunWith(UnitilsJUnit4TestClassRunner.class)

See ya next time!