Default hook state and the test suite

[Vincent Ladeuil]

>>>>> Martin Pool <mbp at canonical.com> writes:

    > On 20 October 2010 09:41, Gordon Tyler <gordon at doxxx.net> wrote:
    >> On 10/19/2010 9:43 AM, John Arbash Meinel wrote:
    >>> On 10/18/2010 6:30 PM, Gordon Tyler wrote:
    >>>> On 10/18/2010 5:35 PM, John Arbash Meinel wrote:
    >>>>> I'm worried about a whole in our testing setup. Specifically, we have at
    >>>>> least one case where by default we have a hook installed at runtime (by
    >>>>> bzrlib), and the test suite defaults to clearing all hooks. It means
    >>>>> that we aren't testing the "stock" behavior. Which was the point of
    >>>>> clearing the hooks (so that plugins wouldn't cause test failures when
    >>>>> they hook in for extra information.)
    >>> 
    >>>> Perhaps selftest should disable plugins unless --enable-plugins is supplied?
    >>> 
    >>> Then you can't run the plugin tests... Which ideally we would get back
    >>> to having all plugin tests run clean as well.

    > I think the distinction is between whether selftest should force
    > plugins off, or whether it should just default to having them off
    > but allow them to be turned back on.

Altenatively we could punt on how we want this kind of control to be
implemented in selftest and just run selftest with or without the
plugins. There is still the question about how we want to tests various
combinations of plugins but starting with none and all would be a good
first step (and will tell us if we need more, which is far from clear
today). The next step may be to run each plugin alone in addition to the
core but even that may be a waste if we get a clean run for all plugins
together.

    > Most times I run tests, I do them with plugins off because I only
    > want to know about things I may have broken in bzr relative to
    > trunk.

Me too. And it's a good way to found about plugin bugs far later than
needed :-/

I *used* to run with all plugins enabled but I had to resign too long
ago.

    > With typical plugins installed, you don't have a clean test suite,
    > even though you may be clean with noplugins and with by running
    > only the plugin tests:

<shameless plug>
I use:

  BZR_PLUGIN_PATH=-site BZR_PLUGINS_AT=loom@${HOME}/.bazaar/plugins/loom

when running the test suite in a loom so I've got this one at least
covered.

There is also BZR_DISABLE_PLUGINS that can help to exclude only some
plugins if they fail at an innaproriate time (like, come one, I'm
*trying* something, I'll get to you later).

</shameless plug>

    > the problem is the interaction between plugins and builtin tests,
    > as discussed earlier in the thread.  Those plugin authors who
    > aren't core developers are probably even less interested in
    > failures in other plugins they may happen to have installed.

Right, the incantation above is exactly what they need, they just have
to replace 'loom' with their plugin of choice.

    >> Perhaps I'm lacking some context here. Whether plugins are
    >> enabled or not seems to me to be a function of who/what is
    >> running the tests. A plugin author would run selftest with
    >> plugins enabled because they want to test their plugin. Babune
    >> would run selftest with plugins disabled because there you're
    >> more concerned with core functionality, right?

babune cares about test passing, period :) *We* (collectively, the whole
bzr community)[1] decide which tests we want to run...

    >> And Babune could even run selftest a second time with plugins
    >> enabled as part of an integration build, using the set of plugins
    >> that are included with the installers.

    > Right.  So is the question just what the default should be?

Again, an alternative answer is: both.

    > I think defaulting to no plugins for selftest would sweep the
    > problem under the carpet.

So far babune does:

  # We want to keep the core plugins only
  export BZR_PLUGIN_PATH=-site

i.e. only the core plugins are covered. It's just a matter of adding
more parameters to the jobs and being to control which *version* of
which plugin is tested (nudge Gordon: that's why I'd prefer to reuse our
already available bzrlib.config ;-).

    > I'd like to either work towards getting a totally clean suite with
    > typical plugins; and/or to coming up with a better idea for how to
    > test the interaction of core and plugins.

That's why I want a clean test suite with only core plugins on all
supported platforms first[2].

          Vincent

[1]: Ok, that's mostly me so far, but don't mix lack of resources with
original intent here, k ? :-D

[2]: But this may be mostly because the results are not presented in the
most convenient way by hudson as we may as well progress on several
paths in parallel.