Default hook state and the test suite

[Danny van Heumen]

I know that when I did some development last year (and I want to do 
more, but am swamped with my study at the moment), I was confused 
because of all the error messages when running the self-test suite. They 
turned out to be caused by plugins. (Plugins that were shipped with the 
OS, so obviously (very) out of date.)

So from a newbie p.o.v. I'd prefer to not automatically test the 
plugins. (Given that they may not even be supposed to work with the 
version being tested. Initially, it was also unexpected that the plugins 
got tested that weren't even available in either the local bzr branch or 
in my own '.bazaar/plugins' directory.)

What about splitting the command into 'selftest-core' and 
'selftest-plugins'? (or some variation thereof) This probably would make 
it more predictable.

-- Danny


On 10/21/2010 03:16 AM, Martin Pool wrote:
> On 20 October 2010 09:41, Gordon Tyler<gordon at doxxx.net>  wrote:
>> On 10/19/2010 9:43 AM, John Arbash Meinel wrote:
>>> On 10/18/2010 6:30 PM, Gordon Tyler wrote:
>>>> On 10/18/2010 5:35 PM, John Arbash Meinel wrote:
>>>>> I'm worried about a whole in our testing setup. Specifically, we have at
>>>>> least one case where by default we have a hook installed at runtime (by
>>>>> bzrlib), and the test suite defaults to clearing all hooks. It means
>>>>> that we aren't testing the "stock" behavior. Which was the point of
>>>>> clearing the hooks (so that plugins wouldn't cause test failures when
>>>>> they hook in for extra information.)
>>>> Perhaps selftest should disable plugins unless --enable-plugins is supplied?
>>> Then you can't run the plugin tests... Which ideally we would get back
>>> to having all plugin tests run clean as well.
> I think the distinction is between whether selftest should force
> plugins off, or whether it should just default to having them off but
> allow them to be turned back on.
>
> Most times I run tests, I do them with plugins off because I only want
> to know about things I may have broken in bzr relative to trunk.  With
> typical plugins installed, you don't have a clean test suite, even
> though you may be clean with noplugins and with by running only the
> plugin tests: the problem is the interaction between plugins and
> builtin tests, as discussed earlier in the thread.  Those plugin
> authors who aren't core developers are probably even less interested
> in failures in other plugins they may happen to have installed.
>
>> Perhaps I'm lacking some context here. Whether plugins are enabled or
>> not seems to me to be a function of who/what is running the tests. A
>> plugin author would run selftest with plugins enabled because they want
>> to test their plugin. Babune would run selftest with plugins disabled
>> because there you're more concerned with core functionality, right? And
>> Babune could even run selftest a second time with plugins enabled as
>> part of an integration build, using the set of plugins that are included
>> with the installers.
> Right.  So is the question just what the default should be?
>
> I think defaulting to no plugins for selftest would sweep the problem
> under the carpet.
>
> I'd like to either work towards getting a totally clean suite with
> typical plugins; and/or to coming up with a better idea for how to
> test the interaction of core and plugins.
>