Test suite failing in buildds - possibly because of zookeeper dying?

[Kapil Thangavelu]

Excerpts from Clint Byrum's message of Thu Jan 12 09:13:19 +0100 2012:
> https://launchpadlibrarian.net/89582471/buildlog_ubuntu-precise-i386.juju_0.5%2Bbzr442-2juju2~precise1~ppa0_FAILEDTOBUILD.txt.gz
> 
> I notice too that it took quite a bit longer than usual. I need some help
> understanding why this is failing.. it doesn't seem to happen in oneiric,
> but its pretty consistent in precise.
> 
> There's also another set of similar failures in natty:
> 
> https://launchpadlibrarian.net/89451958/buildlog_ubuntu-natty-i386.juju_0.5%2Bbzr442-1juju2~natty1_FAILEDTOBUILD.txt.gz
> 
> Notice that on oneiric, the test suite and build of the package only took
> 8 minutes, while the builds above were 28 and 30 minutes respectively.
> 
> https://launchpad.net/~juju/+archive/pkgs/+build/3081196
> 
> Any help is appreciated, thanks.
> 

Per discussion with Clint at the platform rally, we believe this is likely due 
to a problematic buildd environment, that's resulting in a resource constrainment 
that's causing issues for the zookeeper server. The test suite is showing 
several cascading timeouts, but this hasn't been observed anywhere else outside 
of a small percentage buildd servers. The overall size of data in zk for any 
given test run is very small (on the order of a few mb) and is cleared out per test.

cheers,

Kapil