Stable Release Update Regression/Build Problem

[Steve Beattie]

On Tue, Jun 24, 2008 at 07:24:59AM -0400, Scott Kitterman wrote:
> Another related issue is the question of running with *-proposed enabled
> versus installation of single packages from proposed.  This package
> could have only passed verififcation if *-proposed was enabled on the
> system in question.

When we do SRU verifications, we do enable *-proposed, but that's only so
we can pull the specific package that we're trying to test.  We try to be
very careful to make sure that the only additional package we install is
the one we're trying to test; specifically we try to do verification in
pristine vm images and install the specific package via "apt-get install
<package>" just so that we do not get "pollution" from other packages in
*-proposed. For verifying fixes, we specifically try to not do blanket
pulls from *-proposed; the QA team is trying to encourage power users
and developers to run *-proposed to catch embarrassing regressions,
but that's separate from how we want to verify bugfixes.

Sometimes, this approach bites us in other ways; for example, in testing
a fix for the gnome-games package, I spent a couple of hours[0] becoming
convinced that the package in *-proposed didn't fix anything before
I realized that that actual fix was contained in the gnome-games-data
package (built from the same source package). And in fact, I assert that
there's a certain unreality to that approach of verification as well,
as someone who has one package installed will likely have the other
installed and get both when updating, which does not necessarily get
reflected by the way that we're testing.

All that said, the people doing SRU verifications are human and fallible,
and do make mistakes sometimes. It's possible that in installing aptoncd,
the verifier overlooked that the python-central update got pulled in
as well. Improvements to processes and tools and assistance in doing
verifications is greatly appreciated.

> While running with *-proposed enabled may have some benefit for
> regression detection, I don't think it is suitable for SRU verification.
> Generally, I the verification should be against *-release and *-updates
> without other packages from *-proposed.

This is exactly how we try to operate. Finding a clear way to express
this in the wiki documentation would be appreciated.

[0] Admittedly, I was doing this by playing a solitaire variant. See,
    if you help out with SRU verifications, you too can justify playing
    solitaire for two hours as a contribution to improving Ubuntu. :-)

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : https://lists.ubuntu.com/archives/ubuntu-motu/attachments/20080624/51d89e6c/attachment.pgp