Intermittent test failures with juju-core

[William Reade]

On Fri, 2013-02-15 at 14:21 +1300, Tim Penhey wrote:
> I was all ready to say they all passed again, but no... a different test
> failed this time:
> 
> http://pastebin.ubuntu.com/1654670/
> 
> 
> Now I'm sure I don't need to tell you how problematic intermittent
> failing tests are.  Do we have any plans to fix these?

To be fair, roger did work on them as soon as we noticed them, and it's
not always easy to be certain whether an intermittent failure is really
fixed or not. Roger, though, that last one seems to be new; I presume
you're on it. 

> [1] We really should have a protected trunk that normal devs don't have
> permission to write to, but the landing robot (like tarmac) does.

Agreed in principle, but I'm not sure it would have helped us much here.
That's the trouble with intermittent failures...

> [2] launchpad.net/juju-core/worker/uniter	89.354s
> What are we doing here that takes almost a minute and a half?

Most of this is UniterSuite, which takes roughly 80s for me. At least
20s is (apparently non-negotiable) setup/teardown for a totally clean
environment in each of the 35-odd cases; after that it gets a bit harder
to identify serious hotspots. I can see a few ways to speed things up by
a second or two, but I can't see any obviously big wins that don't
involve sacrificing coverage of the phase space of all uniter states.

I'll try a couple of them over the weekend, though, and see what I can
come up with.

Cheers
William