Releasing Alphas and Betas without "freezing"
bryce at canonical.com
Mon Jun 18 17:51:39 UTC 2012
On Mon, Jun 18, 2012 at 11:49:46AM +0200, Rick Spencer wrote:
> On Mon, Jun 18, 2012 at 7:02 AM, Martin Pitt <martin.pitt at ubuntu.com> wrote:
> > Sebastien Bacher [2012-06-15 17:26 +0200]:
> >> Can we just drop the image rolling part of milestones?
> > Our automated tests are still waaaay to incomplete for this step. In
> > manual testing we have found quite a number of real deal-breaker bugs
> > which the automatic tests didn't pick up. We also need to test the
> > current images on a wider range of real iron; which is something our
> > automated QA could do one day, but doesn't right now.
> > So regular manual testing rounds are still required, and the points
> > when we do them might just as well be called "milestones".
> But if the focus is testing, we should optimize the schedule around
> testing. For example, I think Ubuntu would benefit from more frequent
> "rounds" of such in depth testing than the current alpha/beta
> milestones provide. (I think every 2 weeks would be a good cadence).
This could be very beneficial if it were more aggressively organized.
We did something similar with proprietary driver testing one release a
few years back. We had people "join" a team, and then had them install
isos and run through a checklist once a week. I found it to be quite
valuable, but you had to be very organized for it to be useful.
So this wasn't just a "install the image and file bugs" exercise, but an
deliberate look for serious regressions. By having each person provide
a continuous series of data points we could spot anomalies much more
easily. If someone is installing things exactly the same way, on the
same hardware, every week, and all of a sudden one week it fails, that
helps you narrow things down a lot. Or equally important is seeing that
a fix you roll out does indeed restore functionality across multiple
The key was to be very specific in the data collection, else you can
generate a lot of noise quickly. Make a printable survey form they can
fill in as they go through the checklist procedure, and a system info
dumping tool that captures all the logs when they're done that might be
needed for bug reports. The QA team has a tool for capturing all this
data and showing it in a tabulated form so you can spot patterns and
changes over time.
The most important thing is that the data actually get used. This
testing can take a fair bit of time, but if the testers know their
efforts are helping to make things tangibly better they can really get
passionate about doing it.
More information about the ubuntu-devel