Unit Tests & Integration Tests

Fri Sep 12 13:46:34 UTC 2014

I have been trying to digest the following series of talks between Martin
Fowler, Kent Beck, and David Heinemeier, called "Is TDD Dead?
<http://martinfowler.com/articles/is-tdd-dead/>". The topic is a bit
inflammatory, but there's some good stuff here.

Some major points discussed:

   - Is there such a thing as test-induced damage to the architecture?
   - Should you expect your developers to practice TDD (i.e. does that
   workflow work for everyone)
   - Mocks vs. Integration (inside-out vs. outside-in testing)

There's a lot to chew on here. Personally, I agree with what others have
said in this thread: I think like most things in life, you need to find the
middle-way and have both integration and unit tests. To know how healthy
your testing ecosystem is, ask yourself these 3 questions:

   1. Which do I spend more time on: my changes, or writing tests for my
   changes?
   2. Do I need to learn an entirely orthogonal way of interacting with my
   code for testing?
   3. How many defects are getting through?

Enjoying following this thread.

-
Katherine

On Fri, Sep 12, 2014 at 5:42 AM, Michael Foord <michael.foord at canonical.com>
wrote:

>
> On 12/09/14 06:05, Ian Booth wrote:
>
>>
>> On 12/09/14 01:59, roger peppe wrote:
>>
>>> On 11 September 2014 16:29, Matthew Williams
>>> <matthew.williams at canonical.com> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> There seems to be a general push in the direction of having more
>>>> mocking in
>>>> unit tests. Obviously this is generally a good thing but there is still
>>>> value in having integration tests that test a number of packages
>>>> together.
>>>> That's the subject of this mail - I'd like to start discussing how we
>>>> want
>>>> to do this. Some ideas to get the ball rolling:
>>>>
>>> Personally, I don't believe this is "obviously" a good thing.
>>> The less mocking, the better, in my view, because it gives
>>> better assurance that the code will actually work in practice.
>>>
>>> Mocking also implies that you know exactly what the
>>> code is doing internally - this means that tests written
>>> using mocking are less useful as regression tests, as
>>> they will often need to be changed when the implementation
>>> changes.
>>>
>>>  Let's assume that the term stub was meant to be used instead of
>> mocking. Well
>> written unit tests do not involve dependencies outside of the code being
>> tested,
>> and to achieve this, stubs are typically used. As others have stated
>> already in
>> this thread, unit tests are meant to be fast. Our Juju "unit" tests are
>> in many
>> cases not unit tests at all - they involve bringing up the whole stack,
>> including mongo in replicaset mode for goodness sake, all to test a single
>> component. This approach is flawed and goes against what would be
>> considered as
>> best practice by most software engineers. I hope we can all agree on that
>> point.
>>
>
> I agree. I tend to see the need for stubs (I dislike Martin Fowler's
> terminology and prefer the term mock - as it really is by common parlance
> just a mock object) as a failure of the code. Just sometimes a necessary
> failure.
>
> Code, as you say, should be written as much as possible in decoupled units
> that can be tested in isolation. This is why test first is helpful, because
> it makes you think about "how am I going to test this unit" before your
> write it - and you're less likely to code in hard to test dependencies.
>
> Where dependencies are impossible to avoid, typically at the boundaries of
> layers, stubs can be useful to isolate units - but the need for them often
> indicates excessive coupling.
>
>
>> To bring up but one of many concrete examples - we have a set of Juju CLI
>> commands which use a Juju client API layer to talk to an API service
>> running on
>> the state server. We "unit" test Juju commands by starting a full state
>> server
>> and ensuring the whole system behaves as expected, end to end. This is
>> expensive, slow, and unnecessary. What we should be doing here is
>> stubbing out
>> the client API layer and validating that:
>> 1. the command passes the correct parameters to the correct API call
>> 2. the command responds the correct way when results are returned
>>
>> Anything more than that is unnecessary and wasteful. Yes, we do need
>> end-end
>> integration tests as well, but these are in addition to, not in place of,
>> unit
>> tests. And integration tests tend to be fewer in number, and run less
>> frequently
>> than, unit tests; the unit tests have already covered all the detailed
>> functionality and edge cases; the integration tests conform the moving
>> pieces
>> mesh together as expected.
>>
>> As per other recent threads to juju-dev, we have already started to
>> introduce
>> infrastructure to allow us to start unit testing various Juju components
>> the
>> correct way, starting with the commands, the API client layer, and the API
>> server layer. Hopefully we will also get to the point where we can unit
>> test
>> core business logic like adding and placing machines, deploying units etc,
>> without having to have a state server and mongo. But that's a way off
>> given we
>> first need to unpick the persistence logic from our business logic and
>> address
>> cross pollination between our architectural layers.
>>
>>
> +1
>
> Being able to test business logic without having to start a state server
> and mongo will make our tests soooo much faster and more reliable. The more
> we can do this *without* stubs the better, but I'm sure that's not entirely
> possible.
>
> All the best,
>
> Michael
>
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140912/2bfe4fd2/attachment.html>