autopkgtests that fail on stderr output

[Martin Pitt]

Hello all,

a common case of a failing autopkgtest for newly introduced packages
is that the test(s) output something to stderr. By the original
autopkgtest definition this is a fatal condition. This is certainly
debatable, but this certainly did help us to find regressions (newly
introduced warnings in test compilations, new CRITICALs in glib-y
test programs and the like).

It's a bit unfortunate that Python's unittest defaults to outputting
the test progress to stderr (i. e. something which is not an error at
all). This can be fixed by a TestRunner config option:

  unittest.main(testRunner=unittest.TextTestRunner(stream=sys.stdout, verbosity=2))

(The verbosity one is unrelated, but rather useful to have, too)

But there are other cases where stderr output is expected, and not
easy to quiesce. For those tests the usual approach has been to call
the actual test with "2>&1" in the debian/tests/mytest wrapper, but in
some cases this is impractical. For these autopkgtest 2.3 [1] now
offers a new feature to consider a test passed even if it produces
stderr output if you add

  Restrictions: allow-stderr

to debian/tests/control.

In other news, this version now allows architecture specific
dependencies in debian/tests/control, i. e. you can use something like
"Depends: elilo-installer [i386]" and your test will not just fail on
dependency installation on non-i386 architectures.

Thanks,

Martin

[1] https://launchpad.net/ubuntu/+source/autopkgtest/2.3
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-devel-announce/attachments/20130722/e2a0e8fd/attachment-0001.pgp>