Intermittent test failures with juju-core

[Gustavo Niemeyer]

On Sun, Feb 17, 2013 at 6:57 PM, Tim Penhey <tim.penhey at canonical.com> wrote:
> [1] On Launchpad, if a test failed intermittently it was disabled
> immediately and a critical (or high) bug filed.  Most often it was a
> poor test, being one that relied on specific timings, but IIRC some were
> just parts really hard to test reliably and had to be mocked out in
> several places.

It is not okay in juju either, but we don't just disable them because
that will more likely cause the bug to stay for longer rather than
being fixed. We've worked on making a number of those tests not
intermittent, and at least once it was a real bug rather than a poor
test. I also personally found that such intermittent bugs aren't
generally hard to fix. It just deserves some attention that is
generally not as fun as coding something new.


gustavo @ http://niemeyer.net