location of non-test test code

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Sat, 2006-07-15 at 10:22 -0500, John Arbash Meinel wrote:
> 
> 
>>> The reason for this is that its IMO hard to know where to look for
>>> support code if its sometimes near the relevant code, and sometimes in
>>> bzrlib/tests.
>>>
>>> Separately, but related, I've long desired to split our tests out across
>>> the bzrlib namespace, putting them under the thing they test - i.e.
>>> bzrlib.transport.tests for transport generic tests,
>>> bzrlib.transport.sftp.tests for sftp specific tests, etc.
>> I would rather have a mirrored directory inside the tests dir. So we
>> would have:
>>
>> bzrlib.tests.transport
>> bzrlib.tests.transport.sftp
> 
> We can do this, but I dont see it as an improvement on what we have now.
> 
>> etc.
>>
>> I think grouping the tests has value. But I would like to see them split
>> up more. (It would make it possible/easier to run just part of the tests).
> 
> Probably because I dont get this - how does it make it easier? (I find
> it trivial to run any subset I want these days).

In my wild imaginings, we would have a way to automatically determine
which test packages need to be run, so that it defaults to only running
the set that is expected to be effected. With an option to run
everything. Rather than our current "run everything, unless I specify a
specific subset".

By the way, what do people think about making our current regex match be
case insensitive? I've taken to doing:

bzr selftest -v "(?i)http"

Which sets the insensitive flag in the regex engine, but it isn't a very
nice thing to always have to type.

> 
>>> So, heres a two part proposal.
>>>
>>> Part 1:
>>>  Lets move all the non-test test code currently in bzrlib.tests to
>>> bzrlib.selftest. If there is a clear home for it that is outside
>>> bzrlib.selftest - i.e. if its os lock specific, then I would encourage
>>> us to put it in lock.py.
>> I really don't like 'bzrlib.selftest'. Maybe 'bzrlib.test_support'?
>>
>> I actually put some helper stuff into the files themselves. But after
>> discussing with Martin, I really don't think it should go there.
> 
> Its a lot more than support logic. Theres the root TestCase classes,
> theres the test runner, the test result object, and more. I think
> test_support is misleading by incompleteness.

Those all sound like support code for running tests. but if you have a
better idea...

> 
>> It really is nicer to keep the code itself simple, and just have a
>> logical place to look for the associated tests. Right now it is hard to
>> grep through Transports to look for a specific function/variable,
>> because you tend to hit on the Transport.Server, which is only used in
>> testing.
>>
>> I really think we're combining too much stuff into a single file as it
>> is. Especially with all the implementations mixed into the same file as
>> the interface, when you are trying to find a specific function, you have
>> to search for the name, and then page up to see if you are in the
>> implementation you *think* you are in.
> 
> If you are referring to branch/repository/working tree here - then sure,
> I agree. And, making workingtree into a package, with modules for
> implementations, works well for me - because we can put
> workingtree.tests in place ;).
> 
>>> Part 2:
>>>  Lets split out the test suite for better locality of reference.
>>>
>>> I'm much more interested in Part 1 than Part 2, so if Part 2 is
>>> problematic, just ignore it ;).
>>>
>>> Rob
>> I would be very happy to move the test support code out of bzrlib.tests.
>> I find it difficult to find the right thing when you have test files
>> intermixed with the support files.
>> I'm -1 on putting them into the files that are being tested. More +-0
>> about putting them next to the files that are being tested.
> 
> I was *never* suggesting putting them in the files that are being
> tested.
> 

Okay. This line:
  "support code if its sometimes near the relevant code"

Made me think think along the lines of 'let's move the test code near
the relevant code', and our current method is to dump it into the same
file, which I really don't like.


>> I really like having 'this directory is where all the functional stuff
>> is going on'. 'this directory is a mirror of the other one, which is
>> where the tests happen'. And if you are looking for the support routines
>> for running the tests, use <this pattern>.
>> I haven't quite solidified what that pattern should be.
> 
> 
> I think we can reach a happy compromise, but need some more details
> hammered out.
> 
> Rob

I agree. I think we are in general agreement. Especially on moving out
support code. I'm not sure how to reach agreement about how to split up
the test suite.

Right now we try for:

  bzrlib/foo.py
is tested by
  bzrlib/tests/test_foo.py

You seem to be suggesting to change:
  bzrlib/foo.py
into
  bzrlib/foo/__init__.py
and then test it with
  bzrlib/foo/tests.py

Which is okay, but right now we have 77 source files in bzrlib/ (and 83
.py files in tests/, interesting...).

Anyway, I don't think we want 77 directories. And I don't *think* we
want to use:

  bzrlib/foo.py
  bzrlib/test_foo.py

Because that makes it too hard to find the real source code.

I honestly can't think of a better layout than our current bzrlib/tests/
directory, which mirrors the source tree. As long as we are consistent,
the tests are reasonably easy to find, but tucked away so they don't get
in the way of looking at the source.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEu4/KJdeBCYSNAAMRAsiYAKDORTb9IkOTlWkQbg36y3BbXSJkXgCfaJRM
SRyDTATV/85MyqS0z8oiBBU=
=FJfe
-----END PGP SIGNATURE-----