selftest performance: landing our code faster and developing quicker.
mbp at canonical.com
Mon Aug 31 08:26:54 BST 2009
2009/8/31 Robert Collins <robert.collins at canonical.com>:
> On Mon, 2009-08-31 at 14:59 +1000, Martin Pool wrote:
>> What I am suggesting is generally avoiding writing tests that use
>> dummy implementations of real classes - mock objects etc. Only do
>> that when the real implementation would be unreasonably expensive or
>> difficult to use.
> I dislike [loath, fear, hate] 'Stub' and 'Mock' objects, because they so
> very very easily lead to interface skew. I still use them, but with
> caution and unease.
> However, I love "Test Doubles". As long as they really are a double,
> tested to behave to the same contract as that which they replace.
I agree. I think there is also a grey area in between, and some of
our tests fall into it, or at least traipse it onto the carpet. For
instance some of the networking or ui-related tests that have very
low-level canned data (ie strings to be fed to stdin, or responses to
network requests) - they are perhaps not strictly Mocks but they're
not really plausible implementations.
One can ask whether there are any cases where the implementation might
be useful outside of the test, even if only in interactive testing,
debugging, or similar developer activites. The more you can get one
that is plausibly useful, the better the tests seem to be - and as a
bonus, you get an actually useful implementation! To extend your
example, a CHKRepository that does no compression at all might not be
something people would generally want to use, but it might be handy
for some kinds of testing or to set an upper bar on performance. It
is qualitatively different to a real Mock repository.
If they really do both work and you do dependency injection in the way
described in that document then you can in at least some cases
triangulate the bug by injecting different dependencies. I would like
to try making more use of that by saying "run everything on an
ntfs-simulating transport." (That runs into trouble at the moment
because of things that need real working directories; we'd need to be
clearer about just what the test wants.)
"Supports the same contract" is a very powerful idea but the problem
is that the contracts are never complete, or never completely adhered
to -- particularly if we assume there are bugs in the code which is
after all the point of testing. This is a bit like the reason why
open source is successful but "software ICs" not so much- having
standards is good but also being able to see and debug the
implementation is better.
More information about the bazaar