Intermittent test failures with juju-core

[Tim Penhey]

On 16/02/13 02:20, William Reade wrote:
> On Fri, 2013-02-15 at 14:21 +1300, Tim Penhey wrote:
>> I was all ready to say they all passed again, but no... a different test
>> failed this time:
>>
>> http://pastebin.ubuntu.com/1654670/
>>
>>
>> Now I'm sure I don't need to tell you how problematic intermittent
>> failing tests are.  Do we have any plans to fix these?
> 
> To be fair, roger did work on them as soon as we noticed them, and it's
> not always easy to be certain whether an intermittent failure is really
> fixed or not. Roger, though, that last one seems to be new; I presume
> you're on it. 

Thank you Roger for looking into it.  I guess working with such an async
system we are more likely than most to hit intermittent failures.

I guess it just surprised me.  I do know that some projects accept a
certain amount of intermittent failures, I was curious to know what our
acceptance level was. [1]

Sorry for any inferred slights.

Tim

[1] On Launchpad, if a test failed intermittently it was disabled
immediately and a critical (or high) bug filed.  Most often it was a
poor test, being one that relied on specific timings, but IIRC some were
just parts really hard to test reliably and had to be mocked out in
several places.