Releasing Alphas and Betas without "freezing"
mcasadevall at ubuntu.com
Tue Jun 19 18:06:14 UTC 2012
-----BEGIN PGP SIGNED MESSAGE-----
On Mon, Jun 18, 2012 at 11:33 PM, Rick Spencer
<rick.spencer at canonical.com> wrote:
> On Tue, Jun 19, 2012 at 12:15 AM, Kate Stewart
> <kate.stewart at canonical.com> wrote:
>> For Alpha1, we did 2 respin sets after the first set was built,
>> based on what the manual testing was finding and trying to get a
>> set of ARM desktop images. (Note: We did not have quantal arm
>> desktop images until the week of alpha 1, and then didn't have
>> them again with the dailies between 6/10-6/14). Having
>> milestones does
>> a focus on the full set of images. Daily images and the
>> automated testing are still mostly focusing on unit tests for the
>> x86 desktop and server images in virtualized hardware, and as
>> Martin says, the manual testing is still finding issues on the
>> real hardware that are causing respins.
> I believe there is widespread agreement on this thread that manual
> testing is good and necessary. I also think there is agreement that
> a faster cadence of complete manual testing than is accommodated by
> our current milestones would be desirable. I think it's fair to
> say that we can move ahead with increasing the frequency of manual
> testing with or without changes to our milestones. I will look to
> the Ubuntu Community team to begin with this, as they don't believe
> they are blocked by any other decisions to be made.
> I think the question on the table is, shall we drop most
> milestones altogether, or adopt a system such as Thierry suggests,
> where we use the most recent "good" daily as the milestone image?
I have serious concerns with removing the milestones. As it
stands, several images, including the vast majority of the ARM images,
only get extensively tested at milestones due to the limited userbase
of the image (specifically, highbank and armadaxp as of right now is
limited to a handful of individuals in the world at the moment).
Many critical issues with ARM (and to a lesser extent x86)
have only been found during milestone testing. Without a set of
defined and organized images for testing, more obscure parts of the
installer simply do not get tested; for instance, how many people are
going to test all possible server configurations or test the installer
with no network.
These scenarios are not common for development, but can and do occur
regularly for many users who install Ubuntu for the first time. During
12.04 development, during milestone testing, three bugs* relating to
both usecases were found to cause the installer to silently fail
midway through installation leaving the user with only a partially
Each milestone represents an opportunity for end-users and QA to test
our images in something more resembling a production environment, and
to test use-cases and recipes that may normally not see a lot of
coverage unless one is explicatively checking for edge cases.
Milestones exist to give the Ubuntu developer community to step back,
and check to make sure nothing important has broken, and to gauge our
progress through a cycle. In addition, they provide a dedicated time
where as a community we step forth and check our images to ensure no
regressions have slipped by.
If we remove the milestones, the only period of extensively and review
the images will be just before release. As such, any regressions that
are found would require a scrambled fix during a period that minimal
archive changes are desired and would both be costly in terms of
development effort, and risky as each final freeze upload always
carries the inherent chance of hosing something important.
Unless the final intent is to ultimately abolish releases all
together and move to a rolling-release model, I don't believe we,
as a community, could successfully ship Ubuntu with its excellent
state of quality assurance without the cycles of alpha and beta images.
* - Relevant bug reports:
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
-----END PGP SIGNATURE-----
More information about the ubuntu-devel