adding unit tests that take a long time

[Katherine Cox-Buday]

On 04/27/2016 09:51 PM, Nate Finch wrote:
> So, this is exactly why I didn't want to mention the nature of the
> test, because we'd get sidetracked. I'll make another thread to talk
> about that specific test.
>
> I do still want to talk about what we can do for unit tests that take
> a long time.  I think giving developers the option to skip long tests
> is handy - getting a reasonable amount of coverage when you're in the
> middle of the develop/test/fix cycle.  It would be really useful for
> when you're making changes that affect a lot of packages and so you
> end up having to run full tests over and over.  Of course, running
> just the short tests would not give you 100% confidence, but once
> you've fixed everything so the short tests pass, *then* you could do a
> long run for thorough coverage.

I believe Cheryl has something like this in the works and will be
sending a note out on it soon.

> This is a very low friction way to increase developer productivity,
> and something we can implement incrementally.  It can also lead to
> better test coverage over all.  If you write 10 unit tests that
> complete in milliseconds, but were thinking about writing a couple
> longer-running unit tests that make sure things are working
> end-to-end, you don't have the disincentive of "well, this will make
> everyone's full test runs 30 seconds longer", since you can always
> skip them with -short.
>
> The only real negative I see is that it makes it less painful to write
> long tests for no reason, which would still affect landing times....
> but hopefully everyone is still aware of the impact of long-running
> tests, and will avoid them whenever possible.

I will gently point out that we were prepared to land a test that takes
~17s to run without discussion. The motivations are honest and good, but
how many others think the same? This is how our test suite grows to be
unmanageable.

I also agree with Andrew that the nature of the test should be the
delineating factor. Right now we tend to view everything through the
lens of the Go testing suite; it's a hammer, and everything is a nail.
Moving forward, I think we should try much harder to delineate between
the different types of tests in the so-called test pyramid,
<http://martinfowler.com/bliki/TestPyramid.html> place like tests with
like tests, and then run classes of tests when and where they're most
appropriate.

This is definitely static analysis and should be run in the same stage
as other static analysis tools.

-
Katherine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160428/8b688af3/attachment.html>