Uniter tests for update-status hook - BLOCKER

[Martin Packman]

On 20/07/2015, Tim Penhey <tim.penhey at canonical.com> wrote:
>
> Earlier today I was investigating this CRITICAL BLOCKER bug:
> 	https://bugs.launchpad.net/juju-core/+bug/1475724
>
> At first I thought that bug was referring to a different one, which I
> fixed by skipping a part of a test that was using chmod to make unit
> enter an error state. I filed a bug for the fixing of the skip:
> 	https://bugs.launchpad.net/juju-core/+bug/1476060
>
> However on deeper looking into the first bug, it seems that it is all
> timing related.

Unfortunately tracking failures in the uniter tests is complicated.
I've seen four named tests fail much more often in recent days:

TestUniterRelations
<http://reports.vapour.ws/releases/issue/559b486d749a566c38b6a7bc>

TestUniterCollectMetrics
<http://reports.vapour.ws/releases/issue/555b70eb749a563f74eb83ec>

TestUniterSendMetrics
<http://reports.vapour.ws/releases/issue/55acde69749a560b02ac8d06>

TestUniterUpdateStatusHook
<http://reports.vapour.ws/releases/issue/55ace932749a56072b3d3637>

However it's hard to manage as we have issues in the suite in general,
and uniter test output is not easy to deal with. The logs are giant,
the actual failure lines tend to be non-informative with the real
cause several screens up in the log, multiple tests have basically the
same problems with common code...

So, I'm not sure what bugs we want to file to track the work to get
master in a good state. As best as I can work out we have:

1) A regression on windows from the 16th, probably from this metrics landing:
<http://reviews.vapour.ws/r/2173/>

2) An earlier regression on all platforms tracked in this bug:
<https://bugs.launchpad.net/juju-core/+bug/1475724>

3) A more general problem with the TestUniterSendMetrics.

I'm not even completely sure which of these your windows test skip
resolves, but I assume the first? Fun, huh.

> [1] We should seriously start thinking how to gate landings on the unit
> tests passing on amd64, ppc, and windwos.

I'd love to gate on the windows tests, but don't want to get yelled
at. Recently, three test runs at 40mins each has not been enough to
get a passing suite reliably, but maybe with this latest batch of
fixes that becomes more reasonable again.

Martin