Defaulting tests internal to the package

[William Reade]

On Mon, Jan 25, 2016 at 2:19 AM, Rick Harding <rick.harding at canonical.com>
wrote:

> I've got to toss another +1 for tests at both levels. One set of tests are
> tests against your contract to outsiders. Another is confidence that your
> internals are resilient. There are a ton of cases I can think of such as
> internal code that validates changes in state, validates various forms of
> input, deals with internal changes to document structure over time.
> Ideally, when an external contract test fails, one of the internal ones
> just blew up to point directly at the culprit within all your internal
> code.
>

I really don't think that "both" is any better a default than "internal".
It's a fallback; a patch for missing coverage when you can't effectively
write external tests.

Certainly, changes to doc structure over time are a prime case where it's
probably reasonable to bend principle. But, for example, all the
(critically important!) state-change-validation tests in the state package
*do* make use of an explicitly-injected transaction runner component [0],
and are much the better for it; and I'm pretty sure that input validation
is generally an important part of a component's external contract, and must
be tested as such (or, if complex, delegated to another injected component
-- and the delegation itself tested).

Yes, it can mean that internal tests need to be kept up to date more as the
> internals change, but even then tests provide another layer of "did you
> cover all these cases in your refactoring".
>

A casual double-check for definitions of "refactoring" reveals
variants of "altering
its internal structure without changing its external behavior". At least,
that's what I generally take it to mean; and I imagine we agree that some
quantity of refactoring is necessary to maintain a healthy codebase. And
sadly, IME, the refactoring that is hard to do is the refactoring that does
not get done; and so the package that is not maintained tastefully [1] is
the package that becomes a hideous overgrown nightmare that slows down all
development that touches it.

And internal tests slow down responsible refactoring efforts by an order of
magnitude, because you can no longer safely change the package and have
automatic confidence that a green bar reflects identical
externally-observable behaviour. The existence of *any* internal test is a
small but real leak in package encapsulation; and any time you make an
internal change without auditing the internal tests for impact, you take a
small but real risk that those tests will have subtly broken. You *can* do
it safely, with care and discipline; but you can't do it *quickly*; and if
you want to stay responsive, you need both.

I feel I should reiterate that I'm not trying to *forbid* internal tests;
but I am trying to show that their cost is much higher than is widely
appreciated -- and so that any heuristic that pushes us to use them broadly
is IMO highly suspect. Maybe a rudimentary design-pattern-style description
is the right way to go: i.e. problem, forces, solution, consequences?

Cheers
William

[0] although I couldn't swear that wasn't export_tested somewhere along the
line... I think the point still stands, at least it's explicit injection
from above
[1] for whatever set of expedient and defensible reasons may have applied
at various times
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160125/a386c847/attachment.html>