testing when there are multiple providers of an interface

[Robert Collins]

On Tue, 2005-12-20 at 14:08 +0100, Johan Rydberg wrote:
> jblack at merconline.com (James Blackwell) writes:
> 
> > That said, there's been some hard core work on both the current (weave)
> > and potentially next (append only known as "knit") format. [...] The
> > recent tests for knit showed a whopping 98% reduction in time for bzr
> > branch -- as in from 80 minutes down to less than 2. Its not quite ready
> > for inclusion in official bzr.dev yet -- it still has some bugs, hasn't
> > been reviewed yet and may or may not have tests.
> 
> Regarding knits and testing;
> 
> It is not obvious how the test suite should be re-worked to support
> testing of several history formats: Should all the tests be run with
> all different history formats?  Probably not.  Should some tests be
> run with all different history formats?  Probably.
>
> The work I have done has mostly been about defining APIs to enable the
> bzrlib core to handle several different history formats.  If we write
> API tests for these new API, and test all history formats with them,
> in theory the rest of the tests should pass.  Other key functionality
> also need per-format testing, such as fetchers, cloners, and upgraders.

Yes. So in general when we look at an interface we have 3 groups of
tests:
 * interface provider tests
 * class specific tests
 * interface using tests

The first group are what define the contract of the interface,.

The second group is essentially internal tests of specific providers, or
of interactions between two providers that may not be exercised by the
contract definition. (I.e. 'fast-path' code that only kicks in when two
instances of something are the same type).

The third group is just code that needs that interface.

So we should run the first test group against all the providers. (We do
this with transports). 

We should run the second group once. (We do this with the remote ui
tests where we test the ui against branch urls that support listing, and
dont support listing respectively. This isn't as clean as it can/should
be yet - we should have two interfaces, one for listable and one for not
listable, and then those remote blackbox tests just become part of the
third group.).

We should run the third group once, against any implementation we
choose, and for comfort/developing new implementations we should be able
to switch the used implementation from the command line.

How to do this?

What we want to do is to be able to write tests once and have them
applied to the different cases above. Martin and I have chatted on the
phone about how to do this, and I've got some rather abstract notes
about general applications of this... Heres a concrete suggestion
though, just for bzr.

First lets clearly define 'interface': An interface is a set of symbols
you can use/call with a specific defined behaviour. That means

For interface clients
---------------------
We want to parameterise the entire library in one hit, to the selected
implmentation for *default* behaviour. As an example, lets consider a
strawman 'branch initialiser' as an interface.

This interface has one method:
def initialise(url):
    """Initialise a branch at url."""

Currently all the code accesses *all* implementations of this interface
by the following api:
bzrlib.branch.Branch.initialise(url, format=None,...)

For minimal impact, we can parameterise this by setting the format in a
bzrlib.branch._private variable rather than hardcoding it in the method.

Then the selftest command can be taught how to change that default and
how to restore it after the run. This will change it for all the code in
the library - code that does not depends on a specific format will just
get it by default.


For tests that require specific implementations
-----------------------------------------------
There should be a way of getting a specific implementation of the
interface. The obvious, clearest way is just to create the thing
directly. For instance if we are testing the copies between two local
file system transports (say this is a fast-copy code path):
transport_from = bzrlib.transport.local.LocalTransport(..)

And this will work as long as there is a way to directly create the
objects needed. Sometimes when you have multiple interfaces interacting,
it can be non obvious how to create what you need: just change the code
to allow that in the cleanest way possible.


For interface providers
-----------------------

This is a little more complex to do nicely because we want to run many
tests with each different provider.
 

Currently we manage this by having one or more TestCase classes that are
*not* TestCases. This means they are not picked up by the test loading
infrastructure.

We then subclass them and mixin the required code to instantiate a
specific interface. (Again see the transport tests for this).

What I'd like to do here is to exclude the files containing interface
tests from the normal test loading. If we load them separately, we can
write a trivial TestClass subclass that alters the default
implementation just before the test runs, and restores it afterwards. In
the first instance this can be hard coded for the interface its
switching between - we can worry about parameterising it later. This is
a decorator for the tests. This is slightly more complex than multiple
subclasses of the tests, but has several benefits: new implementations
of interface will not require altering the tests at all; its easier to
read for code writers.

Specifically:
* Put interface provider tests in separate files.

in tests.__init__.test_suite()
* Dont list them in the testmod_names list, rather list that in
  a new list - until we do parameterisation, that would be (here)
  test_branch_mod_names = ['bzrlib.tests.test_branch_interface']
* load these tests and surround them with a decorator:
  for mod_name in test_branch_mod_names:
    mod = _load_module_by_name(mod_name)
    for test in loader.loadTestsFromModule(mod):
        suite.addTests(ProviderTestAdapter().adapt(test))


ProviderTestAdapter().adapt is a method that returns a TestSuite
containing one test for each implementation (it deep copies the test
once for each implementation).

This approach has the following properties that I care about:
- It does not prevent the use of other unittest extensions such as 
  test order randomisation, running just named tests, et al.
- It will work if you use a non standard test runner as long as they
  call test_suite() to get the test suite, or if they scan for tests, it
  will degrade to the default implementation and run the tests once.
  (I'd like to correct this but theres no -solid- de facto standard for
  finding tests from random subtrees in python at the moment.
- The decoration will let us record the implementation in the reported  
  test name, so -v can show something like:
  (Format4) \
  bzrlib.tests.test_branch_interface.TestClone.test_clone_missing_data
- We dont run the risk of forgetting to update some test suite or 
  another as we get more implementations of things. For instance, once
  we have some listing of providers, for SVNBranch, the branch test 
  suite will run on it automatically. (This has happened in the past).


I will do up a mock up of this today I think, as the need for it seems
to be growing.

Rob


-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20051221/2b6fd13f/attachment.pgp