Releasing Alphas and Betas without "freezing"

[Scott Kitterman]

On Tuesday, June 19, 2012 11:06:14 AM Michael Casadevall wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Mon, Jun 18, 2012 at 11:33 PM, Rick Spencer
> 
> <rick.spencer at canonical.com> wrote:
> > On Tue, Jun 19, 2012 at 12:15 AM, Kate Stewart
> > 
> > <kate.stewart at canonical.com> wrote:
> >> For Alpha1, we did 2 respin sets after the first set was built,
> >> based on what the manual testing was finding and trying to get a
> >> set of ARM desktop images.  (Note: We did not have quantal arm
> >> desktop images until the week of alpha 1, and then didn't have
> >> them again with the dailies between 6/10-6/14).  Having
> >> milestones does
> 
> force
> 
> >> a focus on the full set of images.   Daily images and the
> >> automated testing are still mostly focusing on unit tests for the
> >> x86 desktop and server images in virtualized hardware,  and as
> >> Martin says,  the manual testing is still finding issues on the
> >> real hardware that are causing respins.
> > 
> > I believe there is widespread agreement on this thread that manual
> > testing is good and necessary. I also think there is agreement that
> > a faster cadence of complete manual testing than is accommodated by
> > our current milestones would be desirable.  I think it's fair to
> > say that we can move ahead with increasing the frequency of manual
> > testing with or without changes to our milestones. I will look to
> > the Ubuntu Community team to begin with this, as they don't believe
> > they are blocked by any other decisions to be made.
> > 
> > I think the question on the table is, shall we drop most
> > milestones altogether, or adopt a system such as Thierry suggests,
> > where we use the most recent "good" daily as the milestone image?
> 
> I have serious concerns with removing the milestones. As it
> stands, several images, including the vast majority of the ARM images,
> only get extensively tested at milestones due to the limited userbase
> of the image (specifically, highbank and armadaxp as of right now is
> limited to a handful of individuals in the world at the moment).
> 
> Many critical issues with ARM (and to a lesser extent x86)
> have only been found during milestone testing. Without a set of
> defined and organized images for testing, more obscure parts of the
> installer simply do not get tested; for instance, how many people are
> going to test all possible server configurations or test the installer
> with no network.
> 
> These scenarios are not common for development, but can and do occur
> regularly for many users who install Ubuntu for the first time. During
> 12.04 development, during milestone testing, three bugs* relating to
> both usecases were found to cause the installer to silently fail
> midway through installation leaving the user with only a partially
> configured system.
> 
> Each milestone represents an opportunity for end-users and QA to test
> our images in something more resembling a production environment, and
> to test use-cases and recipes that may normally not see a lot of
> coverage unless one is explicatively checking for edge cases.
> 
> Milestones exist to give the Ubuntu developer community to step back,
> and check to make sure nothing important has broken, and to gauge our
> progress through a cycle. In addition, they provide a dedicated time
> where as a community we step forth and check our images to ensure no
> regressions have slipped by.
> 
> If we remove the milestones, the only period of extensively and review
> the images will be just before release. As such, any regressions that
> are found would require a scrambled fix during a period that minimal
> archive changes are desired and would both be costly in terms of
> development effort, and risky as each final freeze upload always
> carries the inherent chance of hosing something important.
> 
> Unless the final intent is to ultimately abolish releases all
> together and move to a rolling-release model, I don't believe we,
> as a community, could successfully ship Ubuntu with its excellent
> state of quality assurance without the cycles of alpha and beta images.
> 
> * - Relevant bug reports:
> https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985737
> https://bugs.launchpad.net/livecd-rootfs/+bug/985258
> https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/985280

+1

I have to confess that when I threw out the idea of just abolishing the 
milestones way back in this thread I thought it was a sufficiently ridiculous 
idea that it would give people pause about dropping the freezes.

People worried about velocity through a freeze can publish stuff in a PPA and 
ask them to test it during a freeze.  I think this entire notion is going to 
add significant risk to the development cycle.  Michael is right on target.  
Without a dedicated focus on human testing of various components things are 
going to be missed until the end game when broad user testing starts.

Scott K