ppc64le timeouts cursing things

Fri Jul 17 12:08:02 UTC 2015

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17.07.2015 12:07, James Tunnicliffe wrote:
> /me opens can of worms
Thanks for starting the discussion :)

> 
> Having spent perhaps too long trying to parallelise the running of
> the unit test suite over multiple machines using various bat guano
> crazy ideas, I know too much about this but haven't got an easy
> fix. I do know the right fix is to re-write the very long tests
> that we have.
> 
> If you want to find long running tests, go test -v ./... -check.v
> is the command to run at top level. You will get a lot of output,
> but it isn't difficult to find tests that take longer than 10
> seconds with grep and I am sure I could dig the script out that I
> wrote that examines the output and tells you all tests over a
> certain runtime.
> 
> When you run "go test ./..." at the top of juju/juju it runs suites
> in parallel. If you have multiple long tests in a suite then it has
> a significant impact on the total runtime. We have no way with the 
> current tools to exclude single tests without modifying the tests 
> themselves;

How about GOMAXPROCS=1 go test ./... ? Won't that force the runtime to
run all suites sequentially?

> if we did we could run all the tests that take less than a few
> seconds by maintaining a list of long tests, and run those long 
> tests as a separate, parallel task. The real fix is to put some
> effort into making the long running tests more unit test and less
> full stack test. 30+ seconds is not what we want. The least worst
> idea I have is making a sub-suite for tests that take > 10 seconds,
> one test per suite, so the standard tools will run them in parallel
> with everything else. Providing you have many CPUs there is a
> reasonable chance this will help. It is not remotely nice though.

Using go tool pprof can also help figuring out why certain tests take
a long time and/or memory. I'm planning to experiment with it and come
up with some feedback.

> 
> 0 ✓ dooferlad at homework2 
> ~/dev/go/src/github.com/juju/juju/worker/uniter $ go test -check.v
> 
> Shorter tests deleted from this list. The longest are: PASS:
> uniter_test.go:1508: UniterSuite.TestActionEvents 39.711s PASS:
> uniter_test.go:1114: UniterSuite.TestUniterRelations 16.276s PASS:
> uniter_test.go:970: UniterSuite.TestUniterUpgradeGitConflicts
> 11.354s
> 
> These are a worth a look: PASS: uniter_test.go:2053:
> UniterSuite.TestLeadership 5.146s PASS: util_unix_test.go:103:
> UniterSuite.TestRunCommand 6.946s PASS: uniter_test.go:2104:
> UniterSuite.TestStorage 4.593s PASS: uniter_test.go:1367:
> UniterSuite.TestUniterCollectMetrics 4.102s PASS:
> uniter_test.go:774: UniterSuite.TestUniterDeployerConversion
> 6.904s PASS: uniter_test.go:427:
> UniterSuite.TestUniterDyingReaction 5.772s PASS:
> uniter_test.go:393: UniterSuite.TestUniterHookSynchronisation
> 4.546s PASS: uniter_test.go:1274:
> UniterSuite.TestUniterRelationErrors 4.536s PASS:
> uniter_test.go:476: UniterSuite.TestUniterSteadyStateUpgrade
> 6.405s PASS: uniter_test.go:895:
> UniterSuite.TestUniterUpgradeConflicts 6.430s
> 
> ok   github.com/juju/juju/worker/uniter 175.014s
> 
> James
> 
> On 17 July 2015 at 04:59, Tim Penhey <tim.penhey at canonical.com>
> wrote:
>> Hi Curtis,
>> 
>> I have been looking at some of the recent cursings from ppc64le,
>> and the last two included timeouts for the worker/uniter tests.
>> 
>> On my machine, amd64, i7, 16 gig ram, I get the following:
>> 
>> $ time go test 2015-07-17 03:53:03 WARNING juju.worker.uniter
>> upgrade123.go:26 no uniter state file found for unit
>> unit-mysql-0, skipping uniter upgrade step OK: 51 passed PASS ok
>> github.com/juju/juju/worker/uniter      433.256s
>> 
>> real    7m24.270s user    3m18.647s sys     1m2.472s
>> 
>> Now lets ignore the the logging output that someone should fix,
>> we can see how long it takes here. Given that gccgo on power is
>> slower, we are going to do two things:
>> 
>> 1) increase the timeouts for the uniter
>> 
>> 2) change the uniter tests
>> 
>> WRT to point 2, most of the uniter tests are actually fully
>> functional end to end tests, and should not be run every time we
>> land code.
>> 
>> They should be moved into the featuretest package.
>> 
>> Thanks, Tim
>> 
>> -- Juju-dev mailing list Juju-dev at lists.ubuntu.com Modify
>> settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> 

- -- 
Dimiter Naydenov <dimiter.naydenov at canonical.com>
Juju Core Sapphire team <http://juju.ubuntu.com>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJVqPAiAAoJENzxV2TbLzHwIHYIAKLXI2F4V/Jp+3rFLqbOCrgx
QHTnnnARC7yDE5nbz0nFC/Z6JdEIsG+Xc+JzsaYh+cpZiRTmRvwztSlOyFBq649a
fpCyUttY7CvPGxf+ul58dkFD2JL7Pv/ZNOAR4vGS6X2IR5y/UohtJVntkh3i68xQ
+zRNlhmrGs2pxYVTHMPjfO+X83Cv/UNHq/j7upk1jRKXrm4AjjqGS+vQkIvTUJDF
Y2T8efxFXHnMP5u3qI6yyoE1C8/wjh2AHkNNcVPoAy8ClRVjowOo0UpSH8XV2k89
PRtA35ON7Xrgrv45SOehuDo7PyeZacop7wp2d+tNKLLV4xi75aKkt7EQUcfmNOk=
=I+Ar
-----END PGP SIGNATURE-----