[RFC/MERGE](0.17) Auto load tests

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Wed, 2007-05-02 at 18:01 +1000, Ian Clatworthy wrote:
>> Robert Collins wrote:
>>> On Fri, 2007-04-27 at 18:08 -0500, John Arbash Meinel wrote:
>>>   
>>>> Mostly I'm looking for feedback. I wanted to implement this for a
>>>> plugin
>>>> I was writing, because I was getting tired of manually tracking
>>>> filenames. (Also with plugins you have to be careful about your python
>>>> path, because you are actually loaded as bzrlib.plugins.PLUGIN.tests
>>>> not
>>>> PLUGIN.tests). 
>>>>     
>>> I'm personally not a fan of autodiscovery of tests - it just feels like
>>> too much magic. If we are going to do it I'd rather we looked at
>>> leveraging nose or trial than rolling our own. -0 from me.
>>>
>>> -Rob
>>>   
>> Robert,
>>
>> Can you give some more background as to why you dislike autodiscovery of 
>> tests? Are there particular scenarios where you've seen it cause more 
>> problems that it solves?
> 
> All the autodiscovery tools I've seen fail silently. This sucks :(. We
> have a complex (but worth the complexity) test setup, and I'd hate to
> see test coverage shrink due to a defect in the test loader.
> 
> Thats the primary concern for me given that John did allow a package
> boundary to introduce different discovery policies. If I was writing a
> discovery process I'd be doing that at the level of a module not a
> package though, FWIW.
> 
> Rob

Well, right now our tests are broken up by packages. We have
branch_implementations and bzrdir_implementations, etc.

So it seemed a natural boundary. And it means that every module doesn't
have to implement test_suite.

If you would prefer, I could go another route, which would be to check
for test_suite() on every module, and if it is present, use it instead
of directly loading.

Then we have the question of whether the package should override all
sub-modules. There are 2 options

1) All bzrdir_implementanions/* code does:

from bzrlib.tests.bzrdir_implementations import \
  adapt_test_suite_for_bzrdir

test_suite = adapt_test_suite_for_bzrdir

2) The package can override the behavior for all sub-modules.


If we do 2, then we haven't really diverged much.


As for "fail silently". I match based on the name "test_*.py". If it
matches that name, an import failure should be produced if the code
doesn't import properly (rather than squelching the import). If it
doesn't, then I would consider that a bug. (I should probably test it
explicitly).

I didn't change the existing TestLoader() code though. So inside that
module everything has to inherit from unittest.TestCase. If it doesn't,
then it doesn't get loaded. But we already have that constraint.

If I make sure you get nasty tracebacks if your test files fail to load,
does that handle your concern?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGP7e5JdeBCYSNAAMRAt3EAKDNiU/mtT2TUw5FEATxvINp0LpccQCggHuY
Ai6E2BxNnix9vuXgbILk5fg=
=q9t3
-----END PGP SIGNATURE-----