test scenario adaption

[Vincent Ladeuil]

>>>>> "martin" == Martin Pool <mbp at canonical.com> writes:

    martin> On Thu, Jun 5, 2008 at 5:02 PM, Vincent Ladeuil <v.ladeuil+lp at free.fr> wrote:
    >> Err, I really prefer *this* form or any form capturing the test
    >> parameters in a single place in a single class with subclasses
    >> tweaking the scenarios at will.
    >> 
    >> If you look at tests/test_http.py you'll see that you can then
    >> decide test class by test class what parametrization you want to
    >> apply.
    >> 
    >> Here you separate between all_repository_format_scenarios() and
    >> formats_to_scenarios(), but using adapter classes leave far more
    >> room for variations.

    martin> Hello Vincent,

    martin> Thankyou for reading this too.

    martin> Parameterized tests are pretty powerful but I think
    martin> we are all going at them in slightly different ways.
    martin> So maybe we can try again to settle a standard way,
    martin> and avoid confusion or debating it for each patch.

    martin> It seems to me that what api clients might need to do are

    martin>  1- provide, under a name, a set of scenarios: "all repository formats"
    martin>    1a- combine together two sets of scenarios to get the cross
    martin> product: "all (repository, branch) pairs"
    martin>    1b- filter the scenarios by a condition like "all repository
    martin> formats that support external references"
    martin>  2- specify that some set of tests should be run once for each
    martin> scenario in a named set; this might be all tests in a class or in a
    martin> module.

Sounds good.

    martin> Maybe other things need to be on that list?

Nothing obvious without re-reading some code except that care
should be taken to avoid generating duplicate ids for tests.

    martin> I believe that people will not normally need to
    martin> customize _how_ the tests are adapted: just putting a
    martin> dictionary onto the test object lets it do all
    martin> necessary adaption during setup or construction.

Agreed.

    martin> Anyhow, in each of those places I'd like to make the
    martin> api say them as straightforwardly as possible, to
    martin> make it easy for people to add new parameterized
    martin> tests.  Ideally it would hide the construction of
    martin> TestSuites, the construction of test instances, use
    martin> of TestLoaders and so on because as far as I can see
    martin> none of these tests have a good reason to care about
    martin> changing them.  And ideally the two of them would be
    martin> decoupled for clarity and so that if someone wants to
    martin> reuse that list of scenarios they can do so.

I need to page in all the related modules, so the following are
just from memory.

The problem I had was mainly that the existing framework was
making it hard to use a specialized loader and my patch addressed
that.

Another problem was that some scenarios was "shared" between
modules using adapters (bundles and transport_implementations I
think).

    martin> Now it seems to me that the first one basically needs
    martin> to generate a sequence of dicts and a function is
    martin> enough for that. As Robert says, you can compose
    martin> them.  On the other hand classes can be more
    martin> extensible.  But I dislike that TestScenarioAdapter
    martin> mixes together the job of generating this list and of
    martin> multiplying it out across tests.

I think we (Robert you and I) agree that this is where the
confusion started.

    martin>     # XXX: Isn't load_tests() a better way to provide the same functionality
    martin>     # without forcing a predefined TestScenarioApplier ? --vila 080215

    martin> Could you please explain this comment more?

It's related to the confusion above. I missed a way to better
share the scenarios to generate different scenarios and some
classes were already specializing TestScenarioApplier into
*Adapters. I get tricked there and didn't think deeper, the patch
was already pretty invasive for my taste.

And by the way, that comment was a question, thanks for finally
answering it :-P

    martin> Also, there are some older tests that use ad-hoc
    martin> methods of parametrization like subclassing;
    martin> hopefully once we decide about this we can update
    martin> them.

Indeed.

IMHO we agree on the aims and the principle, we just need to code
it :)

My plate being already full, I'll track this thread but make no
promise about doing it myself* :)

But this is an important subject and, once again, illustrate the
difficulty we have in enforcing some modifications throughout the
whole code base. May be we should file bugs for such
incoherences (may be incoherency is too strong a word, what I
mean is that for hysterical raisins several styles co-exist and
make it hard for newcomers to understand which one is the Right
one).

     Vincent

*: But, damn, that's an interesting subject ;-)