Releasing Alphas and Betas without "freezing"
kate.stewart at canonical.com
Mon Jun 18 22:15:52 UTC 2012
On Mon, 2012-06-18 at 11:49 +0200, Rick Spencer wrote:
> On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt <martin.pitt at ubuntu.com> wrote:
> > Sebastien Bacher [2012-06-15 17:26 +0200]:
> >> Can we just drop the image rolling part of milestones? We still
> >> probably want fixed checkpoints in the cycle to review the features,
> >> etc but they don't especially need to be linked with a special
> >> image...
> > Our automated tests are still waaaay to incomplete for this step. In
> > manual testing we have found quite a number of real deal-breaker bugs
> > which the automatic tests didn't pick up. We also need to test the
> > current images on a wider range of real iron; which is something our
> > automated QA could do one day, but doesn't right now.
> > So regular manual testing rounds are still required, and the points
> > when we do them might just as well be called "milestones".
> But if the focus is testing, we should optimize the schedule around
> testing. For example, I think Ubuntu would benefit from more frequent
> "rounds" of such in depth testing than the current alpha/beta
> milestones provide. (I think every 2 weeks would be a good cadence).
Between the 12.04.1 and Quantal Milestones, the QA Testing and QA
Community testing have a pretty full load already. (see columns)
What was decided to try with Quantal was to do a more intense round
of manual testing on the dailies, the week before the milestone,
so that the bugs found could be fixed by development, and still give the
developers a good window of focused transitions and feature development
time. This possibly could be adjusted to a round of testing 2 weeks
prior, but would have to be juggled in with the testing team's other
commitments? We're releasing Beta 1 on 9/6, Beta 2 on 9/27 and Final
on 10/18 - each 3 weeks apart, so not as much room there.
Not sure how many of the other Ubuntu flavors (Kubuntu, Xubuntu, etc.)
that come out with the alpha milestones would want to participate in a
more frequent testing schedule though. They already skip some of the
milestones, based on which of their packages are landing and resources
are available to do the manual testing, but do have an implied
dependency on Ubuntu alpha/betas being available.
For Alpha1, we did 2 respin sets after the first set was built,
based on what the manual testing was finding and trying to get
a set of ARM desktop images. (Note: We did not have quantal arm
desktop images until the week of alpha 1, and then didn't have them
again with the dailies between 6/10-6/14). Having milestones does force
a focus on the full set of images. Daily images and the automated
testing are still mostly focusing on unit tests for the x86 desktop and
server images in virtualized hardware, and as Martin says, the manual
testing is still finding issues on the real hardware that are causing
More information about the ubuntu-devel