Feedback on a base "fake" type in the testing repo

[Gustavo Niemeyer]

On Wed, Feb 11, 2015 at 4:53 PM, Eric Snow <eric.snow at canonical.com> wrote:
> tl;dr Using fakes for testing works well so I wrote a base fake type. [1]
>
> While working on the GCE provider, Wayne and I started taking a
> different approach to unit testing than the usual 1. expose an
> external dependency as an unexported var; 2.export it in
> export_test.go; 3. patch it out in tests. [2]  Instead we wrote an
> interface for the provider's low-level API and implemented the
> provider relative to that. [3]  Then in tests we used a fake
> implementation of that interface instead of the concrete one.  Instead
> of making the actual API requests, the fake simply tracked method
> calls and controlled return values.

This is a "mock object" under some well known people's terminology [1].

There are certainly benefits, but there are also very well known
problems in that approach which should be kept in mind. We've
experienced them very closely in the first Python-based implementation
of juju, and I did many other times elsewhere. I would probably not
have written [2] if I had a time machine.

The most problematic aspect of this approach is that tests are pretty
much always very closely tied to the implementation, in a way that you
suddenly cannot touch the implementation anymore without also fixing a
vast number tests to comply. Tests organized in this fashion also tend
to obfuscate the important semantics being tested, and instead of that
you see a sequence of apparently meaningless calls and results out of
context. Understanding and working on these tests just a month after
you cooked them is a nightmare.

The perfect test breaks if and only if the real assumption being
challenged changes. Anything walking away from this is decreasing the
test quality.

As a recommendation to avoid digging a hole -- one that is pretty
difficult to climb out of once you're in -- instead of testing method
calls and cooking fake return values in your own test, build a real
fake object: one that pretends to be a real implementation of that
interface, and understands the business logic of it. Then, have
methods on it that allow tailoring its behavior, but in a high-level
way, closer to the problem than to the code.

[1] http://martinfowler.com/articles/mocksArentStubs.html
[2] https://labix.org/mocker



gustavo @ http://niemeyer.net