[Bug 1197005] Re: Add "expect stderr" restriction

[Martin Pitt]

Rodney and I discussed that yesterday, and a compromise was to add a new
restriction (or feature) to not count stderr output as a failure.

** Summary changed:

- Printing to stderr should not be considered a test failure
+ Add "expect stderr" restriction

** Changed in: autopkgtest (Ubuntu)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/1197005

Title:
  Add "expect stderr" restriction

Status in “autopkgtest” package in Ubuntu:
  Triaged

Bug description:
  The specification currently says:

      ...if a test exits nonzero, or prints to stderr, it is considered
  to have failed.

  This is backwards. Plenty of programs write status to stderr and then
  exit with a zero status to indicate success. Examples: wget, curl. I'm
  sure there are many others. Unix convention is to use the exit status
  to indicate success/failure, so it would be far more consistent to
  make this behaviour the default.

  This is a problem because now I have to catch every instance of this
  happening and redirect stderr to /dev/null. Worse, this violates the
  principle that when a test fails it should tell me why, since now all
  of stderr which might have told me has been redirected away.

  A workaround might be to capture stderr to a temporary file and then
  only print it if the exit status was non-zero, but this is painful to
  do in every test and there is no good solution for logging
  stderr/stdout ordering (eg. annotate-output's manpage warns about
  this).

  Before the specification is set in stone, IMHO the sense of this
  should be reversed. Tests should exit with a zero status on success.
  If they actually failed because some program failed but returned a
  zero exit status, then that's a bug that should be fixed. It might be
  sensible to define a "restriction" that changes this to the original
  behaviour of writes to stderr being treated as errors, but the default
  should be to use the exit status only.

  If you don't agree, then at the very least please define and implement
  a "stderr-is-not-a-failure" restriction, so capturing stderr and
  handling the exit status does not have to be done in every test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1197005/+subscriptions