Releasing Alphas and Betas without "freezing"

[Adam Conrad]

On Tue, Jun 19, 2012 at 11:06:14AM -0700, Michael Casadevall wrote:
> 
> Milestones exist to give the Ubuntu developer community to step back,
> and check to make sure nothing important has broken, and to gauge our
> progress through a cycle. In addition, they provide a dedicated time
> where as a community we step forth and check our images to ensure no
> regressions have slipped by.

I don't think anyone is arguing that we should do less manual testing.
In fact, I think that milestones create a culture of less testing, in
the sense that people ONLY test during milestones.  What's even worse
is that many people work from milestones as if they're somehow blessed
images that are better than the daily produced two weeks later (we
even have teams that take snapshots of the archive to avoid developing
against a moving target which, again, means they're testing against a
milestone, not the real deal).

I think any discussion of "dropping milestones" can only come paired
with a conversation about better continuous testing practices.  If
people have the time to test, they should test what's current.  If
people don't have the time to test, milestones don't magically create
time, in fact, they drain time from people who have to make them go.

Maybe the above math seems simpler to me than it is to others, I'm
not sure, but if the argument is that we don't have the manpower to
test more, then I don't see how halting several people in their tracks
every once in a while creates manpower.  It doesn't.  It can't.  Now,
one could argue instead that every team should take a few hours a week
to test a daily image, and I'd be all for that.  Of course, if we
pick different teams per day, all the best bugs will get filed on
Wednesday by the kernel team, because they're clearly the best non-QA
image testers we've got. :P

Seriously, though.  There's no way that a process that takes time can
create time.  It's simple physics.  I understand the fear that dropping
milestones could lead to less testing.  And, in a culture where we
only test milestones, that's obviously true.  We need a cultural shift.

The only reason milestones sometimes still look like a "mess" and need
time to settle is because we don't test enough between them.  If we
tested enough between them, we wouldn't need milestones.  We've put
serious (and I'd say quite successful) effort into making the archive
installable and generally sane all cycle with the +1 maint effort, and
I think it's time we extended that culture to a +1 QA culture, where
testing is consistent, weird bugs are filed every day, and we don't
have these 4-day long panic periods where WE HAVE TO FIX EVERY WEIRD
BUG RIGHT NAO 'cause no one bothered to install from a daily for the
last month.

Maybe I'm way off base.  Maybe I'm a nutter.  But I think I see where
Rick's coming from here, and it's not about "muahaha, let's elminate
testing by eliminating milestones", it's "hey, maybe we could be testing
so consistently that milestones become an obvious waste of time".

... Adam

PS: I leave you with a quote from IRC from the last milestone week:
    "most of the bugs we know about, while not good, we can live with
    for an A1, and they'll get fixed on the next daily."