Release testing and the relationship between 'bzr selftest' and plugins

[Vincent Ladeuil]

>>>>> Jelmer Vernooij <jelmer at samba.org> writes:

    > Am 16/03/12 09:22, schrieb Vincent Ladeuil:
    >>>>>>> Jelmer Vernooij <jelmer at samba.org> writes:
    >> <snip/>
    >> 
    >> >> I'd say soft dependencies in bzr-core and build dependencies for
    >> >> packages or is should that be recommendations instead ?
    >> > We have to bundle the build dependencies, otherwise the plugins can't be
    >> > used and thus bundling them becomes pointless (since e.g. PQM will skip
    >> > all the tests).
    >> 
    >> > If we require e.g. PQM to have the dependencies pre-installed then we
    >> > end up with another problem, which is ensuring that the right build
    >> > dependencies are installed.
    >> 
    >> Which we need to address one way or the other if we want to test the
    >> plugins anyway.
    >> 
    >> Now, PQM will have a hard time as the dependencies needs to be installed
    >> and they may change from one bzr series to the other and PQM still needs
    >> to run the tests for all supported series...
    > At that point, there isn't really that much of an advantage in bundled
    > plugins though. We have to have some mechanism of checking out external
    > code, building it and updating anyway. If we're doing that for libapr,
    > libsvn, subvertpy, dulwich, mercurial, qt, pyqt, etc, then we might as
    > well use the same mechanism to obtain a copy of the plugins.

Unless we disable said plugins on pqm and rely on something else to run
the tests... Yeah, brittle, but worth keeping in the back of our minds.

    >> 
    >> Which reminds me that we really want to run the tests for the supported
    >> series instead of just trunk (outside of pqm that is).
    > I thought pqm just ran on lucid?

Well, yeah, it runs on lucid, for now and also with rather old versions
and subunit and testtools. That's slightly worrying but we've been
living with that for quite some time...

<snip/>

    >> Huh ? Do you mean, the plugin authors don't participate enough or that
    >> packagers don't want to use these tricks ?

    > I'm not sure if it's "don't want". I think it's more of a time
    > issue.  Both of these require significantly more time from plugin
    > authors or packagers.

    > E.g. bzr-rewrite, bzr-webdav or bzr-grep don't go and tag each
    > time a new bzr release is out, because it usually just works with
    > each release.

Hehe, glad you mention bzr-webdav which is encountering an issue right
now :)

bzr-2.5 breaks it. So, indeed, I was wondering about two possible
scenarios:

- make one release for bzr < 2.5 (tag it), one release for 2.5 (tag it),
  propose merging into core during the 2.6 cycle.

or

- create one series for up to 2.4 and another for 2.5 and up

Or some mix.

And when trying to cut a release for < 2.5 I did:

   modified      info.py
                                                                        

=== modified file 'info.py'
--- info.py	2011-11-18 15:29:09 +0000
+++ info.py	2012-03-16 15:48:14 +0000
@@ -15,13 +15,13 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
-bzr_plugin_name = 'webadv'
+bzr_plugin_name = 'webdav'
 
 # versions ending in 'dev' mean development version
 # versions ending in 'final' mean release (well tested, etc)
-bzr_plugin_version = (1, 12, 3, 'dev', 0)
+bzr_plugin_version = (2, 4, 0, 'final', 0)
 
-bzr_compatible_versions = [(2, x, 0) for x in [2, 3, 4, 5,]]
+bzr_compatible_versions = [(2, x, 0) for x in [2, 3, 4,]]
 
But no luck ! Since we didn't bump bzrlib.api_minimum_version, this does
not work :-(

    > Requiring packagers to subscribe to the trunk of all the branches

Yes.

    > and notice incompatible changes costs time too.

<cough> \tiny{no, beta testers are supposed to raise this}

    >> qbzr maintain different series.
    > It's one of the notable exceptions though (together with bzr-svn).

\o/

    >> The osx installer (and the windows installer too AFAIK) has a script to
    >> download from a branch (tip) or a tag or a revid, so it's just a matter
    >> of providing either a branch (during betas) or a tag for stable
    >> releases.
    > We still had out of date plugins in the installer, though.

Yeah, the process is still not polished enough :-/

Ideally, at least osx and windows, should share their scripts (or
release definitions) so we can update it in a single place. 

More and people are submitting merge proposals just for bumping which
version of a plugin should be packaged.

I think that's the good trend and we should probably help it becoming
the official way to get packaged.

The testing effort we're talking about here may be another user of such
a release definition or become the validating place for it (dreaming
aloud).

    >> Once this is fixed, the freebsd will regain some color and we can
    >> restart from there.
    > Which subunit/testtools issues are those? Perhaps we can do something to
    > get those fixed.

Unfortunately this is one a subunit/testtools issue, my setup is:

- maintain the right branches for subunit/testotols under ~/src

- export via nfs (cough),

- automount from slaves (cough cough)

The last part started failing for freebsd mounting from precise and... I
switched to other stuff[1] :-/

    Vincent

[1]: For my defense, I can testify that FreeBSD has historically been
one of the stablest slave, so I'm not *that* worried to lose it for a
while ;)