Feedback on a base "fake" type in the testing repo

[Eric Snow]

Thanks for the great feedback, Gustavo!  I have some follow-up inline.
It reflects my understanding of what you wrote, so don't hate me if
any of it reflects a *mis*understanding. :)  If anything I say *seems*
to imply some deeper criticism or veiled aspersion, just assume I
didn't actually mean to imply anything and take my statement/question
at face value. I'm a pretty straight-forward person and I likely just
phrased things poorly. :)

-eric

On Fri, Feb 13, 2015 at 6:00 AM, Gustavo Niemeyer <gustavo at niemeyer.net> wrote:
> On Wed, Feb 11, 2015 at 4:53 PM, Eric Snow <eric.snow at canonical.com> wrote:
>> tl;dr Using fakes for testing works well so I wrote a base fake type. [1]
>>
>> While working on the GCE provider, Wayne and I started taking a
>> different approach to unit testing than the usual 1. expose an
>> external dependency as an unexported var; 2.export it in
>> export_test.go; 3. patch it out in tests. [2]  Instead we wrote an
>> interface for the provider's low-level API and implemented the
>> provider relative to that. [3]  Then in tests we used a fake
>> implementation of that interface instead of the concrete one.  Instead
>> of making the actual API requests, the fake simply tracked method
>> calls and controlled return values.
>
> This is a "mock object" under some well known people's terminology [1].

With all due respect to Fowler, the terminology in this space is
fairly muddled still. :)

>
> There are certainly benefits, but there are also very well known
> problems in that approach which should be kept in mind. We've
> experienced them very closely in the first Python-based implementation
> of juju, and I did many other times elsewhere. I would probably not
> have written [2] if I had a time machine.
>
> The most problematic aspect of this approach is that tests are pretty
> much always very closely tied to the implementation, in a way that you
> suddenly cannot touch the implementation anymore without also fixing a
> vast number tests to comply.

Let's look at this from the context of "unit" (i.e. function
signature) testing.  By "implementation" do you mean you mean the
function you are testing, or the low-level API the function is using,
or both?  If the low-level API then it seems like the "real fake
object" you describe further on would help by moving at least part of
the test setup down out of the test and down into the fake.  However
aren't you then just as susceptible to changes in the fake with the
same maintenance consequences?

Ultimately I just don't see how you can avoid depending on low-level
details ("closely tied to the implementation") in your tests and still
have confidence that you are testing things rigorously.  I think the
best you can do is define an interface encapsulating the low-level
behavior your functions depend on and then implement them relative to
that interface and write your tests with that interface faked out.

Also, the testing world puts a lot are emphasis on branch coverage in
tests.  It almost sounds like you are suggesting that is not such an
important goal.  Could you clarify?  Perhaps I'm inferring too much
from what you've said. :)

> Tests organized in this fashion also tend
> to obfuscate the important semantics being tested, and instead of that
> you see a sequence of apparently meaningless calls and results out of
> context. Understanding and working on these tests just a month after
> you cooked them is a nightmare.

I think this is alleviated if you implement and test against an
interface that strictly defines the low-level API on which you depend.

>
> The perfect test breaks if and only if the real assumption being
> challenged changes. Anything walking away from this is decreasing the
> test quality.

Agreed.  The challenge is in managing the preconditions of each test
in a way that is robust to low-level changes but also doesn't
"obfuscate the important semantics being tested".  In my experience,
this is not solved by any single testing approach so a judiciously
chosen mix is required.

>
> As a recommendation to avoid digging a hole -- one that is pretty
> difficult to climb out of once you're in -- instead of testing method
> calls and cooking fake return values in your own test, build a real
> fake object: one that pretends to be a real implementation of that
> interface, and understands the business logic of it. Then, have
> methods on it that allow tailoring its behavior, but in a high-level
> way, closer to the problem than to the code.

Ah, I like that!  So to rephrase, instead of a type where you just
track calls and explicitly control return values, it is better to use
a type that implements your expectations about the low-level system,
exposed via the same API as the actual one?  This would likely still
involve both to implement the same interface, right?  The thing I like
about that approach is that is forces you to "document" your
expectations (i.e. dependencies) as code.  The problem is that you pay
(in development time and in complexity) for an extra layer to engineer
and maintain, and your tests are less isolated (meaning the scope of
each test increases).  The benefit of codified expectations is roughly
achievable through comments rather than code, but comments have their
own deficiencies, so that also isn't the perfect solution to
communicating expectations about the low-level behavior.

Regardless, as I noted in an earlier message, I think testing needs to involve:

1. a mix of high branch coverage through isolated unit tests,
2. "enough" testing to ensure your expectations for the low-level API are met,
3. "enough" coverage of the full stack (at least common-path) via
integration tests.

Your recommendation on a low-level implementation to use for testing
is a good one (and one I'll make use of), but it's only one piece of
the testing puzzle.  That said, I don't think you point is that it's
the only testing approach one should use. :)  I appreciate you
bringing it up.