Release testing and the relationship between 'bzr selftest' and plugins

[Jelmer Vernooij]

TL;DR: Nothing proves that plugins are compatible with a particular
   version of bzr like running the full testsuite, with all plugins
   installed. I think we should run the testsuite often - with all
   plugins installed - and make sure it passes.

We have recently had a few issues with Windows and Mac packages
bundling plugins that were incompatible. There have been a
fair number of similar issues with packaged (individual) plugins in
Debian/Ubuntu in the past. I'm wondering what we can do to reduce
this skew, and its impact.

The API versioning infrastructure doesn't work
----------------------------------------------

In the past, we've tried to give plugins the ability to declare
what bzr versions they support. The problem with this is that
it only uses a single tuple to represent the entire public bzrlib API,
of which plugins usually only use a tiny amount. We've tweaked the
way this works several times, and it doesn't get in the way too much
at the moment, but it also doesn't really help in detecting
incompatibilities.

If we change the version tuple for every change that could
potentially break backwards compatibility somewhere,
plugin authors are constantly updating the list of bzr versions
their plugin supports. If we only update it for major changes (what
are major changes?) we risk breaking plugins without noticing.

We can use separate version tuples for for different bzr subsystems,
but that is a pain, both to keep up to date in bzrlib and for plugin
authors to check. http://pad.lv/742192

Similarly, plugin authors can't predict whether a future API change is
going to affect their plugin. It's impossible to say which future
bzrlib version is going to break your plugin. http://pad.lv/704238

As somebody (I think poolie?) said earlier, we should probably limit API
version checking in plugins to:

 * If applicable, is the bzrlib version known to be too old
 * If applicable, is the bzrlib version known to be too new

Plugins tightly coupled to bzr core
-----------------------------------

Some plugins have a fairly tight relationship with Bazaar because they
tie into it in so many places. Changing the bzrlib core isn't always
(or at reasonable cost) possible in a way that doesn't break the plugins.
This is especially true of the foreign branch plugins (bzr-svn,
bzr-git, bzr-hg).

It would be also be nice to be able to blacklist certain versions
of plugins that we know we're breaking, when we make changes. This is
bug http://pad.lv/742196

As the maintainer of three foreign branch plugins, I run their
testsuites regularly, and usually notice when there is an incompatible
change in bzrlib.  I think I've done a reasonable job of keeping
versions available that are compatible with all releases.

However, I do a pretty poor job of keeping everybody else informed
about these kinds of fixes, and when a new version should be shipped.
I'll try to be more vocal about new releases, and send more
announcements to the bazaar-announce list.

"bzr selftest" doesn't pass with a standard set of plugins installed
--------------------------------------------------------------------

At the moment, it is not feasible to run "bzr selftest" on a system
with some of the standard plugins installed, without getting lots of
test failures.

Plugins plug into bzr and register new formats, commands and hooks.
This affects other parts of Bazaar. This can break core bzr tests.
Sometimes this is with good reason - tests
that verify the formatting of option descriptions break when plugins
register new commands with invalid descriptions. And sometimes it is
unintentional - a plugin that hooks into status and tweaks its output
can break the 'bzr status' blackbox tests.

I really think we should fix this.

No regular testing of plugins against bzr.dev
---------------------------------------------

"bzr selftest" gets run very often for "bzr" itself with just the bundled
plugins (launchpad, changelog_merge, ...) - on PQM, various platforms
in babune, on developer machines, etc.

However, we infreuqently run the testsuites of plugins against
bzr.dev (usually only when we are actively changing the plugins), and
even less frequently we run the tests outside of
bzrlib.plugins.PLUGINNAME.tests that are relevant for the plugin.

Once we fix the previous issue, I'm sure more developers will start
running more of the tests. Perhaps it would also be possible to
have a babune slave run the tests for all plugin trunks against
bzr.dev?

No 'bzr selftest' run with the bzr and plugins shipped in an installer
---------------------------------------------------------------------

This also related to the previous two points. At the
moment, we don't run the full testsuite, including plugins, for the
combination of bzr and plugins that is going to be shipped in an
installer.

Once we have a working "bzr selftest", this should be easy to do. And
we're less likely to find issues at install time if the full testsuite
is already being run regularly.  Of course, it will slow down the
release process somewhat, having to wait for the full testsuite for
bzr core and all plugins to pass and all.

Thoughts?

Cheers,

Jelmer