[merge] doc how to use new test features

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> On 5/2/07, Aaron Bentley <aaron.bentley at utoronto.ca> wrote:
>> Martin Pool wrote:
>> > silently return - this test just doesn't make sense in this case
>>
>> I don't like silent returns at all.  The test should not report that it
>> passed unless it ran to completion.
> 
> Yes, that was my point in my later post -- and that this is commonly
> how TestSkipped is currently used.  I don't mind if we have a
> different status meaning "test not implemented" or "test prerequisite
> not implemented yet."

Well, one specific example is "test_put()" for readonly transports.
There are several put tests, and *one* of them should test that the
right error is raised, but the rest of the test isn't worth doing.

It seems like a good case for "silent return", because readonly
transports act differently for something like put().


And what about Branch format 0.0.4. We don't support creating these from
scratch (and don't really plan on ever allowing it).

> 
>> > This raises the question of when it's acceptable to have tests in any
>> > of these cases.  Will we merge code with known failures?
>>
>> I think that has to be case-by-case.  We would certainly merge a test
>> case that failed for already-present code.  Otherwise, it's a tradeoff
>> based on the utility of the code vs the importance of the failure vs the
>> difficulty in fixing the code.
> 
> Right.  Maybe we should just disclose it in reviews - "this adds one
> more known failure for bug #123".
> 

Well, we already have known failures on win32 and mac OS X, for a
variety of platform dependent reasons. I can't run the full test suite
on my Mac laptop because a lot of tests fail, and I have to sort through
them to know if it is caused by my new code, or if it is an old failing
test.

I've spent some time to clean up a couple tests, but I need to do the rest.

If we aren't planning on rewriting the OS to support not rewriting
filenames, and our old workaround is showing where it is breaking, I
think it is reasonable to call those tests knownFailure, and get on with
other real work.

I don't think we have to support every possible edge case in all
possible circumstances.

What *I* would like to see is all tests which can pass, passing on PQM.
TestSkipped isn't very clear whether this could have passed, or whether
it will never pass.

Which means that I would like MissingDependency tests to fail on PQM,
since we can set that machine up to have all the dependencies.
(otherwise we may miss testing FTP or SFTP, for example).

I would also prefer to have no KnownFailure's on PQM. We could ask for
no KnownFailures on any platform. But honestly that would mean not
testing certain edge cases. (Only using Unicode filenames that are not
combining characters). Though we could switch it around to only have a
couple of tests which test Unicode combined characters, and the rest of
the tests use "safe" unicode characters.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGOKDWJdeBCYSNAAMRAk4yAKCFLcXz94X9/fIuj2Cixq1Bw5AL3QCfStUI
cQc+t52BStnnJj7hB9277iM=
=cySH
-----END PGP SIGNATURE-----