How can I unit test failure cases?

When writing unit tests, especially when using mock objects, there’s always a risk of falling into the trap of writing tests for the implementation rather than for the API. In my experience the most obvious indication of that is that any refactoring of the code requires major rework of the tests.

Right now I’m sitting with the task of writing tests for a function that promises to return NULL on failure. The function in question allocates memory up to three times, opens two message queues, sends a message on one and expects a reponse on the other. If all goes well it returns the pointer to one of the memory areas allocated. And if it fails, for whatever reason, it returns NULL.

Is it even possible to write a set of tests for it without testing the specific implementation?

I know some of the calls can be re-arranged, e.g. moving all memory allocation to the start of the function, without affecting the caller of the function. However, if I use mock objects to simulate failure in the external libraries and system calls I’d be forced to modify the tests on each such re-arranging of calls. In other words I’m testing the implementation!

Should I think differently about testing failure cases?

Should the API be changed in some way to make unit tests a better fit for testing failures?

⟸ Docker container with ip address on local network, p2 Elm 0.16 with stack ⟹

Jesse McDonald

Here’s one idea: First, stub out the interface, but don’t cause it to fail; just record how many times it’s used. Run the code and verify that it succeeds. Then, for each call to the interface in the first run, repeat the test and make just that one stub call fail. It should be possible to write this genetically so that it’s resistant to changes in the number or order of the calls.

Magnus Therning

@Jesse, yes, that’s a nice idea. Nicer than mine of randomizing failures and making many runs.

In my particular case it’s C so collecting the interface is arguably the most involved bit…

I just can’t believe I’m the first one bumping into this… there ought to be open-source tools out there that handles it!

Leave a comment