test scenario adaption (was Re: [MERGE] repository external reference tests (StackedBranches))

[Martin Pool]

On Thu, Jun 5, 2008 at 5:02 PM, Vincent Ladeuil <v.ladeuil+lp at free.fr> wrote:
> Err, I really prefer *this* form or any form capturing the test
> parameters in a single place in a single class with subclasses
> tweaking the scenarios at will.
>
> If you look at tests/test_http.py you'll see that you can then
> decide test class by test class what parametrization you want to
> apply.
>
> Here you separate between all_repository_format_scenarios() and
> formats_to_scenarios(), but using adapter classes leave far more
> room for variations.

Hello Vincent,

Thankyou for reading this too.

Parameterized tests are pretty powerful but I think we are all going
at them in slightly different ways.  So maybe we can try again to
settle a standard way, and avoid confusion or debating it for each
patch.

It seems to me that what api clients might need to do are

 1- provide, under a name, a set of scenarios: "all repository formats"
   1a- combine together two sets of scenarios to get the cross
product: "all (repository, branch) pairs"
   1b- filter the scenarios by a condition like "all repository
formats that support external references"
 2- specify that some set of tests should be run once for each
scenario in a named set; this might be all tests in a class or in a
module.

Maybe other things need to be on that list?

I believe that people will not normally need to customize _how_ the
tests are adapted: just putting a dictionary onto the test object lets
it do all necessary adaption during setup or construction.

Anyhow, in each of those places I'd like to make the api say them as
straightforwardly as possible, to make it easy for people to add new
parameterized tests.  Ideally it would hide the construction of
TestSuites, the construction of test instances, use of TestLoaders and
so on because as far as I can see none of these tests have a good
reason to care about changing them.  And ideally the two of them would
be decoupled for clarity and so that if someone wants to reuse that
list of scenarios they can do so.

Now it seems to me that the first one basically needs to generate a
sequence of dicts and a function is enough for that. As Robert says,
you can compose them.  On the other hand classes can be more
extensible.  But I dislike that TestScenarioAdapter mixes together the
job of generating this list and of multiplying it out across tests.

    # XXX: Isn't load_tests() a better way to provide the same functionality
    # without forcing a predefined TestScenarioApplier ? --vila 080215

Could you please explain this comment more?

Also, there are some older tests that use ad-hoc methods of
parametrization like subclassing; hopefully once we decide about this
we can update them.

-- 
Martin <http://launchpad.net/~mbp/>