Releasing Alphas and Betas without "freezing"
ubuntu at kitterman.com
Mon Jun 18 22:52:14 UTC 2012
On Monday, June 18, 2012 05:15:52 PM Kate Stewart wrote:
> On Mon, 2012-06-18 at 11:49 +0200, Rick Spencer wrote:
> > On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt <martin.pitt at ubuntu.com>
> > > Sebastien Bacher [2012-06-15 17:26 +0200]:
> > >> Can we just drop the image rolling part of milestones? We still
> > >> probably want fixed checkpoints in the cycle to review the features,
> > >> etc but they don't especially need to be linked with a special
> > >> image...
> > >
> > > Our automated tests are still waaaay to incomplete for this step. In
> > > manual testing we have found quite a number of real deal-breaker bugs
> > > which the automatic tests didn't pick up. We also need to test the
> > > current images on a wider range of real iron; which is something our
> > > automated QA could do one day, but doesn't right now.
> > >
> > > So regular manual testing rounds are still required, and the points
> > > when we do them might just as well be called "milestones".
> > But if the focus is testing, we should optimize the schedule around
> > testing. For example, I think Ubuntu would benefit from more frequent
> > "rounds" of such in depth testing than the current alpha/beta
> > milestones provide. (I think every 2 weeks would be a good cadence).
> Between the 12.04.1 and Quantal Milestones, the QA Testing and QA
> Community testing have a pretty full load already. (see columns)
> What was decided to try with Quantal was to do a more intense round
> of manual testing on the dailies, the week before the milestone,
> so that the bugs found could be fixed by development, and still give the
> developers a good window of focused transitions and feature development
> time. This possibly could be adjusted to a round of testing 2 weeks
> prior, but would have to be juggled in with the testing team's other
> commitments? We're releasing Beta 1 on 9/6, Beta 2 on 9/27 and Final
> on 10/18 - each 3 weeks apart, so not as much room there.
> Not sure how many of the other Ubuntu flavors (Kubuntu, Xubuntu, etc.)
> that come out with the alpha milestones would want to participate in a
> more frequent testing schedule though. They already skip some of the
> milestones, based on which of their packages are landing and resources
> are available to do the manual testing, but do have an implied
> dependency on Ubuntu alpha/betas being available.
> For Alpha1, we did 2 respin sets after the first set was built,
> based on what the manual testing was finding and trying to get
> a set of ARM desktop images. (Note: We did not have quantal arm
> desktop images until the week of alpha 1, and then didn't have them
> again with the dailies between 6/10-6/14). Having milestones does force
> a focus on the full set of images. Daily images and the automated
> testing are still mostly focusing on unit tests for the x86 desktop and
> server images in virtualized hardware, and as Martin says, the manual
> testing is still finding issues on the real hardware that are causing
I think it's also worth mentioning that all this fancy automatic testing
that's supposed to replace milestone testing is only fully planned for
Canonical sponsored flavors. For the rest of us we still have to do manual
testing. I, for one, only do install testing on the milestones (because of
the pressure to get a release tested and out the door). The rest of us are
going to have to do this and it doesn't happen in one day.
I've also been considering how things are supposed to work with incremental
uploads if the development release is always supposed to work. For Kubuntu,
we upload (if we have resources) each KDE SC beta, RC, final, and bug fix
update. The beta releases have bugs. It seems a bit counter to always
wanting a working development release to upload beta releases to it, but
that's how the software gets used and bugs detected and reported so they can
More information about the ubuntu-devel