Write test doubles you can trust using verified fakes

When you’re writing tests for your code you often encounter some complex object that impedes your testing. Let’s say some of your code uses a client library to talk to Twitter’s API. You don’t want your tests to have to talk to Twitter’s servers: your tests would be slower, flakier, harder to setup, and require a working network connection.

The usual solution is to write a test double of some sort, an object that pretends to be the Twitter client but is easier to use in a test. Terminology varies slightly across programming communities, but you’re probably going to make a fake, mock or stub Twitter client for use in your tests.

As you can tell from names like “fake” and “mock”, there’s a problem here: if your tests are running against a fake object, how do you know they will actually work against the real thing? You will often find the fake is insufficiently realistic, or just plain wrong, which means your tests using it were a waste of time. I’ve personally had Python mock objects cause completely broken code to pass its tests, because the mock was a bit too much of a mockup.

There’s a better kind of test double, though, known as a “verified fake”. The key point about a verified fake is that, unlike regular fakes, you’ve actually proven it has the same behavior as the real thing. That means that when you use the verified fake for a Twitter client in a test you can trust that the fake will behave the same as real Twitter client. And that means you can trust your tests are not being misled by a broken fake.

Testing with test doubles

Before looking at verified fakes, let’s take a quick look at why test doubles are useful. Let’s say you have a MessageService class you want to test, and it uses TwitterClient.

MessageServiceTwitterClient

Since you don’t want your tests to talk to the real Twitter API you create a FakeTwitterClient and use it in your tests.

TestsMessageServiceFakeTwitterClient

Now your tests prove MessageService works, but don’t have to talk to the Twitter API… assuming you can trust TwitterClient and FakeTwitterClient behave the same.

In order to make FakeTwitterClient into a verified fake, a fake you can trust, you need to write an additional set of tests that run against both TwitterClient and FakeTwitterClient. These contract verification tests ensure both versions behave the same way; I’ll go into more detail about what they test later on. For now you can just visualize what it looks like:

TwitterContractTestsTwitterClient

TwitterContractTestsFakeTwitterClient

When should you use a verified fake?

A verified fake gives you more assurance that your tests are testing what you think you’re testing, since the test double has been verified to act like the real thing. The downside is that this requires more work, since you have to create an additional set of contract verification tests. That means verified fakes make sense if the API you want to fake is used frequently by test code.

If you e.g. have thirty tests for MessageService where FakeTwitterClient saves you from talking to the real Twitter API you’re likely to benefit from the verified fake. If you only have one test it’s probably not worth the extra work; you should probably just live with a little lack of confidence and not bother writing the contract tests.

How do you write a verified fake?

Let’s say TwitterClient looks like this:

class TwitterClient(object):
    """A client for the Twitter API."""
    def tweet(self, message):
        """Tweet a message for the user."""

    def list_tweets(self):
        """Return a list of the user's tweets."""

This client provides a behavioral guarantees, a contract of sorts: if a message is tweeted it will show up in the list of tweets. You want FakeTwitterClient to provide the same guarantee so that when you can confidently use it as a drop-in replacement for TwitterClient.

You should therefore encode this contract into a test:

def test_tweet_listed(client):
    """A tweeted messages shows up in the list of messages."""
    message = generate_random_message()
    client.tweet(message)
    assert message in client.list_tweets()

And implement a FakeTwitterClient that implements this contract:

class FakeTwitterClient(object):
    """A fake client."""
    def __init__(self):
        self.messages = []

    def tweet(self, message):
        """Tweet a message for the user."""
        self.messages.append(message)

    def list_tweets(self):
        """Return a list of the user's tweets."""
        return self.messages

You now want to run test_tweet_listed twice: once against the real client and once against the fake client, to ensure they both provide the same behavior.

The version that will run against the real client will need to use a real Twitter login (presumably a test account of some sort). This means it will be slow and not something you want developers running regularly. The contract verification test for TwitterClient should therefore be configured to only run on the CI server.

The version of test_tweet_listed that runs against FakeTwitterClient is just another fast in-memory test, so it can be run anywhere by anyone.

You now know both TwitterClient and FakeTwitterClient behave the same. This means you can trust that the tests for MessageService are valid tests even though they rely on FakeTwitterClient and not the real thing.

When are verified fakes insufficient?

TwitterClient may produce errors in some cases, e.g. if a message is too long it might throw a InvalidMessageException. This sort of error can easily be implemented in FakeTwitterClient and verified by the contract tests.

Some errors cannot be verified by a contract test, however. For example, if the Twitter API server has a bug it might return an error that results in an exception being raised by TwitterClient. Or a network error may result in a socket.error exception. The problem with these errors is that they are difficult or impossible to trigger reliably in your contract verification tests.

If you can’t trigger an edge case in your contract verification tests then you don’t have a verified fake for that edge case. The best you can do, if you want to trigger these errors in MessageService tests, is to just go the regular test double route and have an unverified fake or mock.

Verified fakes: a better test double

When you write a fake you are assuming that is somehow behaving the same as the real thing. But code is often buggy, which is why we write tests in the first place. The wrong bug in your fake can mean you’ve invalidated all the tests that use it; you have tests but they tell you nothing useful.

But writing just a little more test code solves this problem. Contract verification tests ensure the fake and the real thing behave the same way. They turn your fake into a verified fake, which means you can actually trust the tests that use it.

Got any questions about verified fakes? Send me an email at itamar@codewithoutrules.com and I’ll be happy to help.


You shouldn't have to work evenings or weekends to succeed as a software engineer. Get to a better place with the Programmer's Guide to a Sane Workweek.