Unit Tests & Integration Tests

Thu Sep 11 19:06:52 UTC 2014

So, I believe unit tests are good things.

   - They tell you that a specific unit of code works as designed, or does
   not.
   - They tell you not just that something broke, but where it broke.
   - They run quickly and can give you a reasonable level of confidence in
   the system very quickly.
   - They encourage well designed boundaries between units, and therefore
   encourage well thought through encapsulation.

Mocking and dependency injection are important tools for creating unit
tests that actually decouple units and allow them to be tested separately.
 These are valuable tools, and the two use cases you describe are reasons
why they are valuable.   But they are not the ONLY reasons why they are
valuable.  There are plenty of others -- performance, test-code
cleanliness/re-use, result granularity, etc.

Like any tools, developers can over-use, or mis-use them.   But, if you
don't use them at all, you often end up with what I call "the binary test
suite" in which one coding error somewhere creates massive test failures.
 In it's most degenerative form (which I have seen several times) there are
thousands of tests which ALL pass or ALL fail, and running the tests gives
you no clear idea of where the failure might have been.

I also believe that integration tests are good things.

   - They tell you that things work in realistic scenarios.
   - They tell you that underlying libraries and system changes don't break
   your code.
   - When clearly written, they can be used as acceptance tests and as a
   tool in communicating functionality.

An offline test runner, run "continuously" that spins up instances of the
system under test, and runs realistic scenarios (even though that takes
time, and presents combinatorial problems) is a valuable tool for making
that happen.  But it doesn't remove the need for small, fast, targeted
tests that give you very rapid feedback while you are developing code.

My belief is that you need both small, fast, targeted tests (call them unit
tests) and large, realistic, full-stack tests (call them integration tests)
and that we should have infrastructure support for both.

On Thu, Sep 11, 2014 at 11:59 AM, roger peppe <rogpeppe at gmail.com> wrote:

> On 11 September 2014 16:29, Matthew Williams
> <matthew.williams at canonical.com> wrote:
> > Hi Folks,
> >
> > There seems to be a general push in the direction of having more mocking
> in
> > unit tests. Obviously this is generally a good thing but there is still
> > value in having integration tests that test a number of packages
> together.
> > That's the subject of this mail - I'd like to start discussing how we
> want
> > to do this. Some ideas to get the ball rolling:
>
> Personally, I don't believe this is "obviously" a good thing.
> The less mocking, the better, in my view, because it gives
> better assurance that the code will actually work in practice.
>
> Mocking also implies that you know exactly what the
> code is doing internally - this means that tests written
> using mocking are less useful as regression tests, as
> they will often need to be changed when the implementation
> changes.
>
> Two good reasons to mock stuff in tests:
>
> 1) because it's awkward or impossible to use the real code in a test
> environment.
> 2) because the real code is slow or unreliable.
>
> When neither of these two criteria apply, I believe we should
> avoid mocking.
>
> As for integration tests, I think we should run them by default,
> but provide a flag for skipping slow tests. The go test command has a
> "-short"
> flag that can be used to skip long-running tests. Perhaps
> we could consider doing something similar (perhaps triggered
> with an environment variable).
>
>   cheers,
>     rog.
>
> > Having integration tests spread about the package and having environment
> > variables that switch on/ off them being run
> >
> > $ JUJU_INTEGRATION go test ./...
> >
> > We could make use of build tags:
> >
> > $ go test -tags integration ./...
> >
> > We could put all the integration tests in a single package:
> >
> > $ go test github.com/juju/juju/integrationtests/...
> >
> >
> > Thoughts?
> >
> > Matty
> >
> > --
> > Juju-dev mailing list
> > Juju-dev at lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140911/1c7412a4/attachment-0001.html>