Defaulting tests internal to the package

Sat Jan 23 15:15:49 UTC 2016

I have two points, which might be biased by my personal opinion here:
1) API end points tests are good, they exercise that what you are
advertising to the public is what you meant, for the cases you are
interested. If they are properly coded you should need to export the least
amount of things in export tests and be able to plug your own stubs through
interfaces.
I don't think that we should call things that exercise a public interface
end to end a unit test though, for me a unit test is a test for the
smallest testable code unit you can manage.
2) all the internal functions adhere to a contract and subject to a set of
assumptions that the developer makes, it is a statement to what are the
boundaries of such a piece of code and a test is a great way to communicate
and maintain that contract. If that contract is broken by third party
dependencies or by a different developer, the best way to find out is to
see a very local test failing instead of knowing that "something" in the
package is broken. I completely disagree on the statement that is false
security, unit tests are not supposed to be mathematical proof just
statistical sampling, I feel that if a test passes for a set of cases I
deem representative of the universe of input my function takes I have a
very justified security on the functionality for the intended purpose and a
solid set of samples for the next person touching that code to respect.
Additionally it is a nice practice that allows people finding corner cases
to add them as test to ensure we won't have regressions while documenting
where in the code the corner case hits.

Sorry for the poor redaction, I wrote this on my phone

On Saturday, 23 January 2016, William Reade <william.reade at canonical.com>
wrote:

> On Sat, Jan 23, 2016 at 12:27 AM, David Cheney <david.cheney at canonical.com
> <javascript:_e(%7B%7D,'cvml','david.cheney at canonical.com');>> wrote:
>
>> I agree with Nate, using external tests just adds more boilerplate.
>>
>
> I agree that it's often *harder* to write external tests; but it's always
> harder to write code that's reliable (and, specifically, here, resilient in
> the face of distracted/hurried maintenance programming, which is a
> perennial problem).
>
> But I'd love to be shown where I'm wrong. Do you have any counterpoints to
> my statements about the practical impact of internal tests?
>
> Cheers
> William
>
>
>> On Sat, Jan 23, 2016 at 7:23 AM, William Reade
>> <william.reade at canonical.com
>> <javascript:_e(%7B%7D,'cvml','william.reade at canonical.com');>> wrote:
>> > On Fri, Jan 22, 2016 at 5:17 PM, Nate Finch <nate.finch at canonical.com
>> <javascript:_e(%7B%7D,'cvml','nate.finch at canonical.com');>>
>> > wrote:
>> >>
>> >> I'm glad to hear Roger's opinion about testing internal code... that's
>> >> exactly how I feel.  True unit tests of small bits of code are easy to
>> >> write, easy to read and understand, and give you confidence that your
>> code
>> >> is doing what you think it'll do in a wide variety of cases.  If the
>> unit
>> >> test fails, it's generally incredibly clear where the fault in the code
>> >> lies, and it's easy to fix either the code or the test.  In 6 months,
>> you or
>> >> someone else can go back to the tests, easily understand what they are
>> >> testing, and modify them trivially when making a bug fix or
>> improvement.
>> >
>> >
>> > Tests for unexported utility functions can give you great confidence in
>> the
>> > operation of those unexported utility functions... but those tests will
>> > continue to pass even if you change the package so that they're never
>> > actually used, and that renders those tests' diagnostic power pretty
>> much
>> > worthless. And, to make things worse, that sort of test is very
>> frequently
>> > used as a justification for skipping tests of the same functionality
>> against
>> > the exported interface.
>> >
>> >>
>> >> While you certainly *can* write code that tests all the corner cases of
>> >> some small utility function through the external API... those tests
>> will
>> >> almost always be incredibly opaque and hard to understand, and
>> generally
>> >> much more complicated than they would be if you could just test the
>> utility
>> >> function by itself.  This is often a problem I have with our current
>> >> tests... it's hard to see what is actually being tested, and even
>> harder to
>> >> verify that the test itself is correct and complete.  When a test
>> fails, you
>> >> often have no clue where the actual problem lies, because the test
>> traverses
>> >> so much code.
>> >
>> >
>> > I think you're misdiagnosing the problems with our current tests. Most
>> of
>> > our rubbish "unit" tests are rubbish because they're not unit tests at
>> all:
>> > they're full-stack tests that are utterly and inextricably bound up
>> with the
>> > implementation details of *all* the layers underneath, and depend
>> critically
>> > upon assumptions about how those other layers work. And, yes, one of the
>> > worst things about those tests is how hard it is to diagnose what's
>> actually
>> > happened when they fail.
>> >
>> > But that's really *not* a good reason to add *more* implicit assumptions
>> > about how various components are connected together, which is exactly
>> what
>> > you're doing when you write tests for unexported implementation
>> details. If
>> > your utility function tests fail, but the package tests don't, all it
>> means
>> > is that you've written bad package tests; or, if your package tests
>> *are*
>> > complete and correct, an isolated utility-test failure just means you've
>> > wasted time writing tests for functionality that delivers no observable
>> > value.
>> >
>> >> Of course, I definitely think you *also* need tests of the exported
>> API of
>> >> a package... but small unit tests are still incredibly valuable.
>> >
>> >
>> > "Small unit tests" are tremendously valuable, I agree. And if your
>> tests for
>> > the exported API are *not* small unit tests, you're Doing It Wrong. But
>> unit
>> > tests for unexported functions are *by definition* tests for
>> implementation
>> > details, not for behaviour, and as such have *negative* value; and not
>> just
>> > because they fossilise the code, or even just because they give you
>> false
>> > confidence, but because they remove any pressure to export a good
>> interface.
>> >
>> > To put it another way: if it's hard to write tests to exercise X
>> internal
>> > functionality, either that functionality is not important... or, worse,
>> it
>> > *is* important but your interface design is making it hard to observe.
>> > That's not a reason to add internal tests: it's a reason to *fix your
>> > interface*. (And *usually* in a juju context that just means "stop
>> screwing
>> > around with globals and make your dependencies explicit". Turns out that
>> > inversion of control is actually a pretty good idea...)
>> >
>> > (Maybe it would help if we had a concrete example? What were the tests
>> that
>> > you felt you needed to pull in-package? It may be that you were
>> originally
>> > talking about a hairy-enough situation that it's worth relaxing the
>> > principles in play here -- even bad tests are often better than *no*
>> tests
>> > -- but you were also advocating that *all* tests should be in-package,
>> and I
>> > really don't think that's a justifiable default.)
>> >
>> > Cheers
>> > William
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev at lists.ubuntu.com
>> <javascript:_e(%7B%7D,'cvml','Juju-dev at lists.ubuntu.com');>
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160123/d96007dd/attachment.html>