Unit Tests & Integration Tests

[Michael Foord]

On 12/09/14 06:05, Ian Booth wrote:
>
> On 12/09/14 01:59, roger peppe wrote:
>> On 11 September 2014 16:29, Matthew Williams
>> <matthew.williams at canonical.com> wrote:
>>> Hi Folks,
>>>
>>> There seems to be a general push in the direction of having more mocking in
>>> unit tests. Obviously this is generally a good thing but there is still
>>> value in having integration tests that test a number of packages together.
>>> That's the subject of this mail - I'd like to start discussing how we want
>>> to do this. Some ideas to get the ball rolling:
>> Personally, I don't believe this is "obviously" a good thing.
>> The less mocking, the better, in my view, because it gives
>> better assurance that the code will actually work in practice.
>>
>> Mocking also implies that you know exactly what the
>> code is doing internally - this means that tests written
>> using mocking are less useful as regression tests, as
>> they will often need to be changed when the implementation
>> changes.
>>
> Let's assume that the term stub was meant to be used instead of mocking. Well
> written unit tests do not involve dependencies outside of the code being tested,
> and to achieve this, stubs are typically used. As others have stated already in
> this thread, unit tests are meant to be fast. Our Juju "unit" tests are in many
> cases not unit tests at all - they involve bringing up the whole stack,
> including mongo in replicaset mode for goodness sake, all to test a single
> component. This approach is flawed and goes against what would be considered as
> best practice by most software engineers. I hope we can all agree on that point.

I agree. I tend to see the need for stubs (I dislike Martin Fowler's 
terminology and prefer the term mock - as it really is by common 
parlance just a mock object) as a failure of the code. Just sometimes a 
necessary failure.

Code, as you say, should be written as much as possible in decoupled 
units that can be tested in isolation. This is why test first is 
helpful, because it makes you think about "how am I going to test this 
unit" before your write it - and you're less likely to code in hard to 
test dependencies.

Where dependencies are impossible to avoid, typically at the boundaries 
of layers, stubs can be useful to isolate units - but the need for them 
often indicates excessive coupling.

>
> To bring up but one of many concrete examples - we have a set of Juju CLI
> commands which use a Juju client API layer to talk to an API service running on
> the state server. We "unit" test Juju commands by starting a full state server
> and ensuring the whole system behaves as expected, end to end. This is
> expensive, slow, and unnecessary. What we should be doing here is stubbing out
> the client API layer and validating that:
> 1. the command passes the correct parameters to the correct API call
> 2. the command responds the correct way when results are returned
>
> Anything more than that is unnecessary and wasteful. Yes, we do need end-end
> integration tests as well, but these are in addition to, not in place of, unit
> tests. And integration tests tend to be fewer in number, and run less frequently
> than, unit tests; the unit tests have already covered all the detailed
> functionality and edge cases; the integration tests conform the moving pieces
> mesh together as expected.
>
> As per other recent threads to juju-dev, we have already started to introduce
> infrastructure to allow us to start unit testing various Juju components the
> correct way, starting with the commands, the API client layer, and the API
> server layer. Hopefully we will also get to the point where we can unit test
> core business logic like adding and placing machines, deploying units etc,
> without having to have a state server and mongo. But that's a way off given we
> first need to unpick the persistence logic from our business logic and address
> cross pollination between our architectural layers.
>

+1

Being able to test business logic without having to start a state server 
and mongo will make our tests soooo much faster and more reliable. The 
more we can do this *without* stubs the better, but I'm sure that's not 
entirely possible.

All the best,

Michael