Default hook state and the test suite

[Martin Pool]

On 20 October 2010 09:41, Gordon Tyler <gordon at doxxx.net> wrote:
> On 10/19/2010 9:43 AM, John Arbash Meinel wrote:
>> On 10/18/2010 6:30 PM, Gordon Tyler wrote:
>>> On 10/18/2010 5:35 PM, John Arbash Meinel wrote:
>>>> I'm worried about a whole in our testing setup. Specifically, we have at
>>>> least one case where by default we have a hook installed at runtime (by
>>>> bzrlib), and the test suite defaults to clearing all hooks. It means
>>>> that we aren't testing the "stock" behavior. Which was the point of
>>>> clearing the hooks (so that plugins wouldn't cause test failures when
>>>> they hook in for extra information.)
>>
>>> Perhaps selftest should disable plugins unless --enable-plugins is supplied?
>>
>> Then you can't run the plugin tests... Which ideally we would get back
>> to having all plugin tests run clean as well.

I think the distinction is between whether selftest should force
plugins off, or whether it should just default to having them off but
allow them to be turned back on.

Most times I run tests, I do them with plugins off because I only want
to know about things I may have broken in bzr relative to trunk.  With
typical plugins installed, you don't have a clean test suite, even
though you may be clean with noplugins and with by running only the
plugin tests: the problem is the interaction between plugins and
builtin tests, as discussed earlier in the thread.  Those plugin
authors who aren't core developers are probably even less interested
in failures in other plugins they may happen to have installed.

> Perhaps I'm lacking some context here. Whether plugins are enabled or
> not seems to me to be a function of who/what is running the tests. A
> plugin author would run selftest with plugins enabled because they want
> to test their plugin. Babune would run selftest with plugins disabled
> because there you're more concerned with core functionality, right? And
> Babune could even run selftest a second time with plugins enabled as
> part of an integration build, using the set of plugins that are included
> with the installers.

Right.  So is the question just what the default should be?

I think defaulting to no plugins for selftest would sweep the problem
under the carpet.

I'd like to either work towards getting a totally clean suite with
typical plugins; and/or to coming up with a better idea for how to
test the interaction of core and plugins.

-- 
Martin