Releasing Alphas and Betas without "freezing"

Tue Jun 19 18:06:14 UTC 2012

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Jun 18, 2012 at 11:33 PM, Rick Spencer
<rick.spencer at canonical.com> wrote:
> On Tue, Jun 19, 2012 at 12:15 AM, Kate Stewart 
> <kate.stewart at canonical.com> wrote:
>> 
>> For Alpha1, we did 2 respin sets after the first set was built, 
>> based on what the manual testing was finding and trying to get a
>> set of ARM desktop images.  (Note: We did not have quantal arm 
>> desktop images until the week of alpha 1, and then didn't have
>> them again with the dailies between 6/10-6/14).  Having
>> milestones does
force
>> a focus on the full set of images.   Daily images and the
>> automated testing are still mostly focusing on unit tests for the
>> x86 desktop and server images in virtualized hardware,  and as
>> Martin says,  the manual testing is still finding issues on the
>> real hardware that are causing respins.
> 
> I believe there is widespread agreement on this thread that manual 
> testing is good and necessary. I also think there is agreement that
> a faster cadence of complete manual testing than is accommodated by
> our current milestones would be desirable.  I think it's fair to
> say that we can move ahead with increasing the frequency of manual
> testing with or without changes to our milestones. I will look to
> the Ubuntu Community team to begin with this, as they don't believe
> they are blocked by any other decisions to be made.
> 
> I think the question on the table is, shall we drop most
> milestones altogether, or adopt a system such as Thierry suggests,
> where we use the most recent "good" daily as the milestone image?
> 

I have serious concerns with removing the milestones. As it
stands, several images, including the vast majority of the ARM images,
only get extensively tested at milestones due to the limited userbase
of the image (specifically, highbank and armadaxp as of right now is
limited to a handful of individuals in the world at the moment).

Many critical issues with ARM (and to a lesser extent x86)
have only been found during milestone testing. Without a set of
defined and organized images for testing, more obscure parts of the
installer simply do not get tested; for instance, how many people are
going to test all possible server configurations or test the installer
with no network.

These scenarios are not common for development, but can and do occur
regularly for many users who install Ubuntu for the first time. During
12.04 development, during milestone testing, three bugs* relating to
both usecases were found to cause the installer to silently fail
midway through installation leaving the user with only a partially
configured system.

Each milestone represents an opportunity for end-users and QA to test
our images in something more resembling a production environment, and
to test use-cases and recipes that may normally not see a lot of
coverage unless one is explicatively checking for edge cases.

Milestones exist to give the Ubuntu developer community to step back,
and check to make sure nothing important has broken, and to gauge our
progress through a cycle. In addition, they provide a dedicated time
where as a community we step forth and check our images to ensure no
regressions have slipped by.

If we remove the milestones, the only period of extensively and review
the images will be just before release. As such, any regressions that
are found would require a scrambled fix during a period that minimal
archive changes are desired and would both be costly in terms of
development effort, and risky as each final freeze upload always
carries the inherent chance of hosing something important.

Unless the final intent is to ultimately abolish releases all
together and move to a rolling-release model, I don't believe we,
as a community, could successfully ship Ubuntu with its excellent
state of quality assurance without the cycles of alpha and beta images.

* - Relevant bug reports:
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985737
https://bugs.launchpad.net/livecd-rootfs/+bug/985258
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985280

Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP4L+RAAoJEHM+GkLSJHY507UP/3G1IOSwI3zotWDi4Kr4mbkg
PtlL2ml7gig4wv7c46ZcWHH9xYj0hbFPxBJDWQrzTVVRQCrKEOscWmV/AAO7kLtO
CdBg9ifI15SWwuvKXvuWPqQLhSe0IcVnZ+BSb7eVq+iouOY1Vw/vPdFMeBxEZc1P
+hMa//9eggQKQalnjX6O57Be2TTZt3+o8k8lQ9GFjnxrezpw6XGT2IpiXh/l0L6D
+eQRgzf7HhDqYDeqsaLPDJSS4fVdtvhFS3C5G1aJW8t4YI5WqccgIRuYwQQ9QwFd
XSzp77nYw5dlnufQsQYS5Hm6KQaEaxfvLkUNaY5dTC1vtvoeZQQhpo/CGnji5bqK
6Bl5/CzYg15O7H8xzcnDYWvbnc8f0cQzwfzIFQFKkuWftdM3D3Oim8MTU6dumXUW
qQssOqkTmxHpljI8m7TuybcjPR+/JnKXHVkNt8xT/hs89DL4XN4AZVU9wLjC1Cl3
shxhPVNI7qLlM0H267f2RBhx5MeEa9Lrz9loaGsrXFgtAxO5lqG548UaAWo90zrx
fHmETYZhGQIUVNk1llyJ6xjpvr1SSOAp3FHYg0G43mQmH09EUxM/S+Gr4UFlZ7sw
JOJObzMNfQ8akLt0mJUJpazWbpriXi24053M+M0QtmdIflcIIPbs8DLmaLSLZ0YB
p7l1IO0P1ztRnHUyaS3s
=QrQF
-----END PGP SIGNATURE-----