adding unit tests that take a long time

[Nate Finch]

So, this is exactly why I didn't want to mention the nature of the test,
because we'd get sidetracked. I'll make another thread to talk about that
specific test.

I do still want to talk about what we can do for unit tests that take a
long time.  I think giving developers the option to skip long tests is
handy - getting a reasonable amount of coverage when you're in the middle
of the develop/test/fix cycle.  It would be really useful for when you're
making changes that affect a lot of packages and so you end up having to
run full tests over and over.  Of course, running just the short tests
would not give you 100% confidence, but once you've fixed everything so the
short tests pass, *then* you could do a long run for thorough coverage.

This is a very low friction way to increase developer productivity, and
something we can implement incrementally.  It can also lead to better test
coverage over all.  If you write 10 unit tests that complete in
milliseconds, but were thinking about writing a couple longer-running unit
tests that make sure things are working end-to-end, you don't have the
disincentive of "well, this will make everyone's full test runs 30 seconds
longer", since you can always skip them with -short.

The only real negative I see is that it makes it less painful to write long
tests for no reason, which would still affect landing times.... but
hopefully everyone is still aware of the impact of long-running tests, and
will avoid them whenever possible.

On Wed, Apr 27, 2016 at 9:12 PM Katherine Cox-Buday <
katherine.cox-buday at canonical.com> wrote:

> That is an awesome idea! +1
>
>
> On 04/27/2016 05:51 PM, Andrew Wilkins wrote:
>
> On Thu, Apr 28, 2016 at 1:44 AM Nate Finch <nate.finch at canonical.com>
> wrote:
>
>> I was actually trying to avoid talking about the test itself to keep
>> things shorter ;)
>>
>> The test is parsing the entire codebase under github.com/juju/juju to
>> look for places where we're creating a new value of crypto/tls.Config
>> instead of using the new helper function that I wrote that creates one with
>> more secure defaults.  There's not really any getting around the fact that
>> parsing the whole tree takes a long time.
>>
>
> The nature of the test is important here: it's not a test of Juju
> functionality, but a test to ensure that we don't accidentally use a TLS
> configuration that doesn't match our project-wide constraints. It's static
> analysis, using the test framework; and FWIW, the sort of thing that Lingo
> would be a good fit for.
>
> I'd suggest that we *do* organise things like this separately, and run
> them as part of the "scripts/verify.sh" script. This is the sort of test
> that you shouldn't need to run often, but I'd like us to gate merges on.
>
> Cheers,
> Andrew
>
>
>> On Wed, Apr 27, 2016 at 1:25 PM Nicholas Skaggs <
>> nicholas.skaggs at canonical.com> wrote:
>>
>>> This is a timely discussion Nate. I'll avoid saying too much off the
>>> top, but I do have a question.
>>>
>>> On 04/27/2016 12:24 PM, Nate Finch wrote:
>>> > I just wrote a test that takes ~16.5 seconds on my machine.
>>> Why does the test take so long? Are you intending it to be a short /
>>> small scoped test?
>>>
>>> Nicholas
>>>
>> --
>> Juju-dev mailing list
>> Juju-dev at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
>
>
> --
> -
> Katherine
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160428/e084fb63/attachment.html>