Release testing and the relationship between 'bzr selftest' and plugins

[Vincent Ladeuil]

>>>>> Jelmer Vernooij <jelmer at samba.org> writes:

<snip/>

    >> I'd say soft dependencies in bzr-core and build dependencies for
    >> packages or is should that be recommendations instead ?
    > We have to bundle the build dependencies, otherwise the plugins can't be
    > used and thus bundling them becomes pointless (since e.g. PQM will skip
    > all the tests).

    > If we require e.g. PQM to have the dependencies pre-installed then we
    > end up with another problem, which is ensuring that the right build
    > dependencies are installed.

Which we need to address one way or the other if we want to test the
plugins anyway.

Now, PQM will have a hard time as the dependencies needs to be installed
and they may change from one bzr series to the other and PQM still needs
to run the tests for all supported series...

Which reminds me that we really want to run the tests for the supported
series instead of just trunk (outside of pqm that is).

    >> >> 2 - push plugin authors to create series targeted at bzr releases: avoid
    >> >> many maintenance issues :) This will also help installer
    >> >> builders/packagers.
    >> > For most plugins, this doesn't scale with the number of release series
    >> > and the size of the plugins. It isn't worth the effort to maintain
    >> > separate release series if it's trivial to be compatible with more
    >> > versions of bzr.
    >> 
    >> Balance to be found again, some plugins may just want to tag specific
    >> revisions for a given series if they don't evolve a lot between series.
    >> 
    >> > For plugins that are tightly coupled with particular bzr versions,
    >> > like the foreign branch plugins, this is an option. But it still
    >> > wouldn't have prevented the problems we had with the 2.5
    >> > installers. Changes between beta 4 and beta 5 broke the foreign
    >> > branch plugins, and the installers shipped with an outdated
    >> > version of those plugins (from the correct release series).
    >> 
    >> Sure, but at least the packagers can subscribe to the tip of a given
    >> branch and be done.
    > In practice neither of these seem to happen though.

Huh ? Do you mean, the plugin authors don't participate enough or that
packagers don't want to use these tricks ?

qbzr maintain different series.

The osx installer (and the windows installer too AFAIK) has a script to
download from a branch (tip) or a tag or a revid, so it's just a matter
of providing either a branch (during betas) or a tag for stable
releases.

<snip/>

    >> 
    >> As in tackling https://bugs.launchpad.net/bzr/+bugs?field.tag=babune and
    >> https://bugs.launchpad.net/bzr/+bugs?field.tag=selftest you mean ?

    > Hah, thanks! Didn't realize you had filed those. We should also
    > address the tests by fixing them or disabling them (and opening
    > bugs).

Right, 'and opening bugs' indeed, with a decorator as the one mentioned
below this could reduce the workload enough to turn it into a good
habit.

    >> > I think we should either fix or disable tests (and file bugs) with
    >> > spurious failures rather than keeping them enabled and stumbling
    >> > over them constantly.
    >> 
    >> > Tests that flap aren't useful for either PQM or CI, I don't think we
    >> > should treat them differently.
    >> 
    >> Right, we had enough of them to decorate them may be ? I did exclude
    >> tests on babune at one point but this is not a good solution as I forgot
    >> about them at one point so we need some in-core tracking to get a better
    >> visibility.
    >> 
    >> Probably something along the lines of re-trying once and warns if it
    >> fail twice but don't let selftest itself fail and emit a final summary
    >> mentioning the number of such spurious failures.
    > Doing this just for tests that are known to be spurious you mean, or for
    > all tests ? I wouldn't be in favor of the latter.

Eeerk, only for the spurious yes :)

<snip/>

    >> I think we agree far more than we disagree on most of the topics so
    >> let's address the ones we agree on ;)
    > Works for me ! :-) So, let's:

    >  * Run "bzr selftest" and file bugs for issues we encounter
    >  * Fix said bugs
    >  * Run "bzr selftest" while we sleep
    >  * Run "bzr selftest" during lunch break
    >  * Run "bzr selftest" in the shower
    >  * ...
    >  * Can we run "bzr selftest" with a set of passing plugins installed on
    > babune? We can start with just one and add more as we verify they pass
    > the testsuite

Yeah, that's what I was working on the last time I worked on babune bug
fall into the rabbit hole encountering issues with subunit/testools that
are currently mounted from my home directory instead of being properly
deployed, the need is roughly the same (all slaves needs some deployment
step before a job is run, the hacks I had in place start breaking when I
upgraded to precise).

Once this is fixed, the freebsd will regain some color and we can
restart from there.

          Vincent