[rfc] list tests known to fail on a platform

[Martin Pool]

It looks like we're making some progress towards getting the test
suite passing on all platforms.  I talked a bit with Ian, Aaron and
John yesterday morning about getting some kind of ratchet that will
catch test regressions on Windows or Mac.

There is a bit of a chicken and egg problem that it's hard to enforce
the test suite passing if it doesn't currently pass, but it's hard to
get everything passing if there is no automatic protection against new
things being broken.  (Either because of actual portability bugs in
Bazaar, or because of tests being added or changed to have platform
dependencies.)

One way out of this would be to say that some set of tests are known
to fail on particular platforms.  Much like KnownFailure at the
moment, if they started unexpectedly passing we'd trap this and it'd
be a clue to either investigate and perhaps mark them as expected to
pass in future.  If we had that up to date at least it would give us a
reference point to work from.

Of course there is some risk that we'd just be sweeping the parrot
under the carpet and never come back to fix them.  There are at least
some xfail tests at present that have sat for a long time, and some
that are kind of miscategorized.  But at least it would let us get a
red/green indication of whether the number is getting worse or better.

If we did want to do this, one option is to add it into the particular
tests that are known to fail, as for example test_bundle does on os x,
just calling knownFailure if it does fail or is expected to.  Another
would be to add a global list of tests that are known to fail on a
particular platform.  If there are many (on Windows maybe?) this might
be more manageable.

-- 
Martin <http://launchpad.net/~mbp/>