[rfc] list tests known to fail on a platform

[Vincent Ladeuil]

>>>>> "martin" == Martin Pool <mbp at canonical.com> writes:

    martin> It looks like we're making some progress towards
    martin> getting the test suite passing on all platforms.  I
    martin> talked a bit with Ian, Aaron and John yesterday
    martin> morning about getting some kind of ratchet that will
    martin> catch test regressions on Windows or Mac.

Solaris ? I don't use Solaris anymore but at one point I made
some attempts to make the test suite run there and was very
close.

But I will, from now on, run the test suite on OSX on a regular
basis.

    martin> There is a bit of a chicken and egg problem that it's
    martin> hard to enforce the test suite passing if it doesn't
    martin> currently pass, but it's hard to get everything
    martin> passing if there is no automatic protection against
    martin> new things being broken.  (Either because of actual
    martin> portability bugs in Bazaar, or because of tests being
    martin> added or changed to have platform dependencies.)

Sure, But at the very minimum test failures and errors should be
checked for each release. I was quite surprised that OSX had
failures for so long (not throwing stones here, I consider myself
partly responsible).

    martin> One way out of this would be to say that some set of
    martin> tests are known to fail on particular platforms.
    martin> Much like KnownFailure at the moment, if they started
    martin> unexpectedly passing we'd trap this and it'd be a
    martin> clue to either investigate and perhaps mark them as
    martin> expected to pass in future.  If we had that up to
    martin> date at least it would give us a reference point to
    martin> work from.

And from there, the effort should be aimed at defining *features*
and keep sys.platform tests in osutils as much as possible.

WorkingTree.case_sensitive, CaseInsensitiveFilesystemFeature and
UTF8Filesystem are good examples, even if file systems are not
platforms.

    martin> Of course there is some risk that we'd just be
    martin> sweeping the parrot under the carpet and never come
    martin> back to fix them.

Far less likely for TDD addicts as we are :-) Minimizing selftest
output should be high on everybody TODO list :) Hence getting
XFAIL messages will constantly remind us of the pending problems.

Nevertheless, mentioning bug numbers in the messages will both
allows synthetic messages and provide tracking.

    martin> There are at least some xfail tests at present that
    martin> have sat for a long time, and some that are kind of
    martin> miscategorized.  But at least it would let us get a
    martin> red/green indication of whether the number is getting
    martin> worse or better.

    martin> If we did want to do this, one option is to add it
    martin> into the particular tests that are known to fail, as
    martin> for example test_bundle does on os x, just calling
    martin> knownFailure if it does fail or is expected to.



    martin> Another would be to add a global list of tests that
    martin> are known to fail on a particular platform.

I have no opinion about making that list global or even
versioned, but *I* keep a list of OSX failing test and use it
with:

  bzr selftest --load-list failures.tests

so that I don't have to run the full test suite every time.

   Vincent