[rfc] list tests known to fail on a platform

[Martin Pool]

On Fri, Sep 5, 2008 at 12:14 PM, Mark Hammond <mhammond at skippinet.com.au> wrote:
> Hi Martin,
>
>> There is a bit of a chicken and egg problem that it's hard to enforce
>> the test suite passing if it doesn't currently pass, but it's hard to
>> get everything passing if there is no automatic protection against new
>> things being broken.  (Either because of actual portability bugs in
>> Bazaar, or because of tests being added or changed to have platform
>> dependencies.)
>
> For my interest, how would the PQM enforce the test suite passes across
> platforms?

If we wanted PQM to enforce it the best option seems to be to give pqm
ssh access to a Windows machine.  It can then copy across the merged
tree, and run the tests there.

It doesn't necessarily need to be done for every single pqm landing;
even if there was a separate buildbot-type-thing running on Windows
there is a big difference between it always having many failures
(which people tend to ignore) and generally having only known bugs.

I think if we get the number fairly low we can encourage people just
to manually test it and report problems.

> If a bug number was associated with and printed for each of the tests that
> were supressed it might help keep things on the radar.  One or 2 tests
> already do that - eg:

We should check all of them have a bug number.

>
> blackbox.test_info.TestInfo.test_info_locking     XFAIL
> 553ms
>    OS locks are exclusive for different processes (Bug #174055)
>
> Which interestingly enough looks like the exact same issue causing the
> LockContention failures.

I was going to say this is a poor use of xfail, but on consideration I
think it's just a poor message.  We can't change that OS locks work
this way.  The problem really is that our format uses them to guard
access to trees of this format.

For existing formats we can't change the locking protocol and so it's
not worth reporting that they have this limitation.  However I guess
it is an ongoing bug that there is no current format that doesn't use
OS locks.

> I think we can get the number of failures small enough on Windows to manage
> individually...

That would be great.

-- 
Martin <http://launchpad.net/~mbp/>