[MERGE] test framework distinguishes skips

[John Arbash Meinel]

Martin Pool wrote:
> On  6 Jul 2006, Aaron Bentley <aaron.bentley at utoronto.ca> wrote:
>>> Nice function, but maybe not quite the right name.  How about
>>> @test_requires or @test_dependency?
>> Yeah, I did think about that.  It's a bit awkward, because the parameter
>> is an exception, e.g. ParamikoNotPresent.  I thought
>> @dependency_failure(ParamikoNotPresent) made more sense than
>> @test_requires(ParamikoNotPresent).
>>
>> But the exception does actually name the missing dependency, so it would
>> be possible to do @test_requires("paramiko").  Would you prefer that?
> 
> Or even just skip_on_exception(ParamikoNotPresent)?

I probably like this one the most. But it may be too generic. While I
realize that ParamikoNotPresent is a dependency exception, the decorator
 looks like it is useful for any exception that would cause you to skip.

I'm not against a decorator like that. But the point was to narrow down
the focus, and make it clear that we are skipping because of a missing
dependency.

Maybe just:
skip_on_dependency_exception(ParamikoNotPresent)

Or
skip_on_exception(ParamikoNotPresent, dependency=True)

Then we parameterize TestSkipped to include *why* the test was skipped.
Which can be 'dependency missing', 'unicode failure', etc.

I also kind of which this type of decorator would let you pass in a
message string to give more information:

skip_on_exception(ParamikoNotPresent, msg='Test requires paramiko')

But if we can include all that information in the parameters, we don't
need another parameter for msg.

> 
>> Even expected failures should be more visible than they are now.  I only
>> recently discovered that one of my machines was skipping a bunch of
>> tests because its console is ISO-8859-1.
> 
> It'd be good to have 0 skips when running it on a typical developer's
> machine, where we can expect to have all the dependencies installed.
> Then if something does start to skip we can look into why.
> 
> However we may end up with disjoint sets of things which can only be
> tested on particular platforms.  Separating the skips as you have should
> help a lot: personally I would like to run the suite with the constraint
> of "it's ok to skip things that require another OS, but everything else
> must pass."  (Hm, there may still be restrictions about default encoding
> or locale, but perhaps they'd be ok in utf-8?)

Well, the full test suite runs with 'LANG=C', which causes lots of
things to skip because of Unicode issues.

Right now, I believe all tests are run if you have a Linux box with
LANG=*.UTF-8

Except for things that are skipped because they are adapted to formats
that don't support them. (Old-format branches that can't be initialized,
LocalTransport doesn't have 'bogus_url' support, and really can't, etc).

But those shouldn't really be TestSkipped, since they are invalid tests.

However, for a while, I think SFTP didn't have bogus_url support, and it
really should (as well for ftp, etc).
So those should be TestSkipped() indicating that we should fix them.
While LocalTransport should be a TestNotPossible().

John
=:->


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060707/4212b416/attachment.pgp