adding unit tests that take a long time

[Eric Snow]

On Wed, Apr 27, 2016 at 8:51 PM, Nate Finch <nate.finch at canonical.com> wrote:
> I do still want to talk about what we can do for unit tests that take a long
> time.

If it takes a long time then it probably isn't a unit test. :)

>  I think giving developers the option to skip long tests is handy -
> getting a reasonable amount of coverage when you're in the middle of the
> develop/test/fix cycle.  It would be really useful for when you're making
> changes that affect a lot of packages and so you end up having to run full
> tests over and over.  Of course, running just the short tests would not give
> you 100% confidence, but once you've fixed everything so the short tests
> pass, *then* you could do a long run for thorough coverage.
>
> This is a very low friction way to increase developer productivity, and
> something we can implement incrementally.  It can also lead to better test
> coverage over all.  If you write 10 unit tests that complete in
> milliseconds, but were thinking about writing a couple longer-running unit
> tests that make sure things are working end-to-end, you don't have the
> disincentive of "well, this will make everyone's full test runs 30 seconds
> longer", since you can always skip them with -short.

Support (both technical and organizational) for explicitly
distinguishing short/long tests in the code would be great.

Further, I would like it if tests could be distinguished with a bit
more granularity than short/long.  It's often helpful to divide tests
into short/medium/long.  We've had a number of discussions on this
list to that effect.  There are other categories of tests that would
be worth segregating like this, such as CI tests (as we move them into
the core repo), other feature tests, and high-level acceptance tests.

In support of that idea, I proposed something along these lines a
while back (the patch didn't go anywhere, mostly because I had other
things to get done):

  http://reviews.vapour.ws/r/1647/

Regardless, I'm in favor of leveraging the existing tooling (e.g. "go
test -short") right now.  I also don't see why going that route would
preclude something later like what I proposed in that patch.  Maybe
I'll dust it off and make it compatible with "go test -short"... :)

>
> The only real negative I see is that it makes it less painful to write long
> tests for no reason, which would still affect landing times.... but
> hopefully everyone is still aware of the impact of long-running tests, and
> will avoid them whenever possible.

I think that's a lesser issue we can deal with later if it becomes a
problem, which I don't think it would. :)

-eric