[merge] doc how to use new test features

[Martin Pool]

On 5/3/07, John Arbash Meinel <john at arbash-meinel.com> wrote:
> > Yes, that was my point in my later post -- and that this is commonly
> > how TestSkipped is currently used.  I don't mind if we have a
> > different status meaning "test not implemented" or "test prerequisite
> > not implemented yet."
>
> Well, one specific example is "test_put()" for readonly transports.
> There are several put tests, and *one* of them should test that the
> right error is raised, but the rest of the test isn't worth doing.
>
> It seems like a good case for "silent return", because readonly
> transports act differently for something like put().

Yes, I think I've done that too.

> And what about Branch format 0.0.4. We don't support creating these from
> scratch (and don't really plan on ever allowing it).

For now I might just say that TestSkipped is appropriate when you want
to make it visible that the test was not run in this situation, and
leave it up to developer judgement whether that is true.  Maybe the
answer is to have a specific TestNotApplicable as andrew says.

(With this and ian's patch doc changes seem to provoke a discussion
about precisely how things are or ought to be, which is good, but it's
also good to just incrementally improve the docs...)

> What *I* would like to see is all tests which can pass, passing on PQM.
> TestSkipped isn't very clear whether this could have passed, or whether
> it will never pass.

Right, so by documenting this and having the other exception classes
I'd like to make that distinction.

> Which means that I would like MissingDependency tests to fail on PQM,
> since we can set that machine up to have all the dependencies.
> (otherwise we may miss testing FTP or SFTP, for example).
>
> I would also prefer to have no KnownFailure's on PQM. We could ask for
> no KnownFailures on any platform. But honestly that would mean not
> testing certain edge cases. (Only using Unicode filenames that are not
> combining characters). Though we could switch it around to only have a
> couple of tests which test Unicode combined characters, and the rest of
> the tests use "safe" unicode characters.

Agree.  So we should add selftest options saying whether any of these
occurrences are taken to mean overall failure.

https://bugs.launchpad.net/bzr/+bug/111914

-- 
Martin