Intermittent test failures with juju-core

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-02-18 5:35, Gustavo Niemeyer wrote:
> On Sun, Feb 17, 2013 at 6:57 PM, Tim Penhey
> <tim.penhey at canonical.com> wrote:
>> [1] On Launchpad, if a test failed intermittently it was
>> disabled immediately and a critical (or high) bug filed.  Most
>> often it was a poor test, being one that relied on specific
>> timings, but IIRC some were just parts really hard to test
>> reliably and had to be mocked out in several places.
> 
> It is not okay in juju either, but we don't just disable them
> because that will more likely cause the bug to stay for longer
> rather than being fixed. We've worked on making a number of those
> tests not intermittent, and at least once it was a real bug rather
> than a poor test. I also personally found that such intermittent
> bugs aren't generally hard to fix. It just deserves some attention
> that is generally not as fun as coding something new.
> 
> 
> gustavo @ http://niemeyer.net
> 

I will say that Tim's set up seems to trigger them more than I've seen
myself. The tarmac bot that lands code to goose should be running the
juju-core tests, and I haven't seen an intermittent failure there. And
I haven't seen an intermittent failure locally.

I've had a fair number of failures (RPC tests hanging, the Mongodb
hangs before upgrading to 2.2.0, etc).

But at least for me, I haven't had failures where I felt if I ran the
test suite again, it would pass.

I certainly agree that intermittent tests break your trust in the test
suite which is bad, and are particularly terrible if you are using a
bot since that can reject perfectly good patches.

However, trying to second guess the test suite leads to the other
mistake where things really are broken, but you land it anyway.

So while I think it is great to catch things and fix them,
intermittent tests have an indirect effect of eroding confidence in
the test suite which is probably significantly worse than the benefit
of having them (IMO).

I have some work together to have tarmac landing patches to juju-core
and if that goes through, I'd imagine we'd be stricter about
intermittent tests.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlEhx+AACgkQJdeBCYSNAAOJzwCgz1mzuRUmhPCYFgLVKPHKC1kY
5XYAn0cYm8TDhIFC+7Ks0B1qtN9Bu60l
=EiJf
-----END PGP SIGNATURE-----