Releasing Alphas and Betas without "freezing"

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/20/2012 12:07 AM, Rick Spencer wrote:
> I think this was a very productive discussion. We considered a lot
> of possibilities from a lot of angles.
> 
> All told, I think there are four points under discussion. I'd like
> to tease them out so we can move forward.
> 
> Question 1: shall we stop freezing the archive at milestones? I
> believe there is not 100% consensus on this point, but enough 
> support to try it for Alpha 2, a la Theirry's suggestion. QA
> Team/Foundations Team, do we/will we have the tools in place for
Alpha 2?
> 

After much thought, I'm strongly disagree with discontinuing freezes
in general. To successfully build images requires a constant and
stable base else images can be trivially skewed due to an ill-timed
upload. This is a fundamental aspect of the image mastering process,
and as such, we break a fundamental assumption of creating stable
images. While its not as obvious on x86/amd64, both ARM and powerpc
have slower buildds and as such have missed daily images before due to
the archive
entering an uninstallable state.

This skew exists from the moment a single architecture completes a
package build to whenever the other four finally catch up. For any
given package, it is within the realm of possibility that any of the
five architectures in Launchpad could be the first to complete a build
and upload a package; for some packages such as libreoffice which have
large amounts of arch:all packages and associated post-processing
work, amd64 will regularly complete the build and upload to Launchpad
before i386*

While the largest of these issues have been solved (arch-any/all
skew), this underlying problem still exists, and as such we have no
way without freezes to ensure a consistent image across all
architectures. While the archive should never be in a state of
uninstallability, there is no guarantee that doing an apt-get install
will net the same package versions across architectures.no additional
QA resources behind it. In addition, for packages with
inter-arch:any-all relations, until the i386 build of a package is
complete, apt will install the older version in-archive until the
builds are complete and the skew is resolved.

By abolishing freezes, effectively alpha 2 would only become a
specific daily image with no special preparation work behind it.

For instance, lets assume a new unity version is uploaded that fixes a
critical crashing bug when a user does X, Y or Z is released the day
before Alpha 2 release. Under the current system, the desktop team
would request a freeze exception, and given the nature of the bug, it
would be accepted, and then all images would be respun to include it,
ensuring that all images have consistent package versioning across
them.

Without the freeze process, it is possible (and in the case of ARM,
even likely) that the unity would not finish building before the final
respin, creating a skew between package versions on the alpha 2
images. When people go to run the alpha 2 images, they would find that
ARM still exhibits the crash behavior even though the image was spun
after the upload that fixed it.

Alternatively, it could also be that the respin triggers before
publisher has finished, and thus none of the images would include the
unity crashing bug fix. During freezes, known issues are collected and
placed on the release notes for those adventurous to dive into Ubuntu
alpha images. With a constantly changing base, creating such a list
would be in my opinion close to impossible.

> Question 3: shall we increase the rate of manual testing? This
> question also arose in the thread. I think there is widespread 
> consensus that we should do this, and it is not actually related
> to the other questions. Community Team, is it feasible to increase
> the rate of full manual testing runs to every 2 weeks or similar?
> 

There's another aspect to this question that has to be considered.
Several images such Kubuntu, Edubuntu, and Xubuntu, and a number of
ports (armel, armhf+omap and the entirety of the powerpc port) are not
supported by Canonical and depend solely on community efforts for full
and proper test coverage.

As of writing, all these images has and have continued to meet the
standards required by the current release process as to be included as
part of the normal Ubuntu development cycle, and base their QA and
testing efforts off the existing milestone process.

Before any changes should be made that would radically alter the
testing dynamics are made, these groups must be consulted and their
input solicited to ensure that they are first willing, and second are
able to meet the demands that bi-weekly testing.

> Question 4: shall we keep snapshots of the development release so
> that we can "bisect" more easily and find when bugs were
> introduced? This question also arose, and also is not tied to the
> other questions. QA Team, is it feasible to keep a set of snap
> shots somewhere for
this purpose?
> 

I personally would welcome this change, or at the very least, the
ability to reconstruct older images 1:1 via jigdo or another similar
tool. As it stands, Librarian and launchpad retain all older debs, so
modifying jigdo so it can simply reconstruct an image based off
librarian debs would go a long way toward achieving goal.

That being said, due to its nature jigdo only works with alternates at
the moment as old squashfs's aren't retained.

* - libreoffice build times on Launchpad as of the most recent upload
(1:3.5.3-0ubuntu1):
i386:  9 hours, 51 minutes, 33.0 seconds
amd64: 4 hours, 31 minutes, 59.1 seconds

Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP4ioUAAoJEHM+GkLSJHY5i4gP/2k/p5r14pU1UAiMhW6jWRLc
KdovnqvV4UuVkfHe+w46JClvPjYh62OX1DH/N8EI57Kt3JAuaeFv00NjWg2Wy1Xc
vESYnNCOjEVYcAHWMasekJdpesmPl0MyAwBSZVJvB2zWDbeaduIxI3J4xy4fHtnC
J6F3ypAn+Yuga1Qc+dXZvIVzIUVWmMyO/beDle/6VWXjOTtQdWNhypoqGbqvQTmF
gRJfmYYpu9xkqlDQ7I6TzVyI6ti/IWC5yNQemswfQoQ3m3zR+mNdDM05Ai2+oriy
MOiKWCJmrno09oKzaHi3BfePaLxMYf8/Z7BKr+hpFEA5Rxnod6m7SaE1PPZLj4lx
ZmufBVlUIiwv92Le28gZVNOd4XbkuZfASf0LUQbsROYCW31dw2PK7DlPI1ESncCn
hMPPspDz+mhdBH7aKyprUX7SM8zMVRL9ZCDgAThdXBeYrUT3n74K0TqAbUonSrhk
+Uir7CB+gzTBpFTs9jdY56Gxv1M/Udmgdq6Zs0dIPWKgLzJGwsOWE6BxGMFP/Z2u
QZ6RkMHg0LNDVIfbRjyPJ3oXtRbkCWbIPvY/2G6UeHp52/lJHHs/6FHh7M555gug
eHkO1j+IWV48vMVHfydIYAr9H3c3PjxeklIqJaXFtkLMQIvM+0uI+zdom6IS7bMB
+OlG59zBtH73Qg4RRBrG
=CLDM
-----END PGP SIGNATURE-----