Daily builds are now back to green, need better test output

Dmitrijs Ledkovs dmitrij.ledkov at ubuntu.com
Wed Nov 6 04:30:48 UTC 2013


I've committed a few fixes to fix test-suite errors on PPAs:

* set & use XDG_RUNTIME_DIR, if not set as otherwise spawning session
init was failing

* which uncovered that test_umask failure, too restrictive umask under
test (remember session-init needs to create subdirectories)

* and disabled timing tests, since those don't work on virtualised builders

So at the moment this brings all daily builds back to green. Let's try
to keep those up =)

WTR declaring failing tests, skipped, expected fail, and bailing out.

At the moment, we mostly abort and/or skip executing tests
with/without telling the user about it. I think it would be best to
implement TAP[1] output for our tests, such that we can properly skip
tests, or mark them as expecting to fail.

Announcing that "some tests may or may not fail, because one has
overlayfs mountpoint" is not helpful. Saying that "test_conf test
#3-7" are expected to fail with a reason due to overlayfs mountpoint,
is much better.

In addition it should enable us to write tests, which are known to
fail at the moment and mark them as TODO for future.

Similarly this will improve ability to report and compare test-suite
results on different platforms (e.g. how many tests are not run in
virtualised PPAs vs nonvirt vs overlayfs).


[1] http://en.wikipedia.org/wiki/Test_Anything_Protocol



More information about the upstart-devel mailing list