Releasing Alphas and Betas without "freezing"

[Scott Kitterman]

On Wednesday, June 20, 2012 04:57:19 AM Adam Conrad wrote:
> On Tue, Jun 19, 2012 at 11:06:14AM -0700, Michael Casadevall wrote:
> > Milestones exist to give the Ubuntu developer community to step back,
> > and check to make sure nothing important has broken, and to gauge our
> > progress through a cycle. In addition, they provide a dedicated time
> > where as a community we step forth and check our images to ensure no
> > regressions have slipped by.
> 
> I don't think anyone is arguing that we should do less manual testing.
> In fact, I think that milestones create a culture of less testing, in
> the sense that people ONLY test during milestones.  What's even worse
> is that many people work from milestones as if they're somehow blessed
> images that are better than the daily produced two weeks later (we
> even have teams that take snapshots of the archive to avoid developing
> against a moving target which, again, means they're testing against a
> milestone, not the real deal).

I do ISO/upgrade testing at milestones because that's the priority for that 
period in the development cycle.  If there wasn't a milestone release and the 
related social pressures associated with it, I'd never do it (It's a PITA and 
takes time from other non-Ubuntu things I can't really spare on a regular 
basis).

I agree with your point about people having faith in the ISOs for too long, 
OTOH, at least it's a stable point to install from.  The later images may be 
better, but they may not.  I think what we mostly need is more people running 
the development release day to day so that actual application bugs get 
detected, upstreamed, and fixed earlier in the cycle.  From that POV it doesn't 
matter if one installs from a daily or from an earlier milestone image and 
immediately does a massive dist-upgrade.

> I think any discussion of "dropping milestones" can only come paired
> with a conversation about better continuous testing practices.  If
> people have the time to test, they should test what's current.  If
> people don't have the time to test, milestones don't magically create
> time, in fact, they drain time from people who have to make them go.

In my experience is does create time because it makes it a priority.

> Maybe the above math seems simpler to me than it is to others, I'm
> not sure, but if the argument is that we don't have the manpower to
> test more, then I don't see how halting several people in their tracks
> every once in a while creates manpower.  It doesn't.  It can't.  Now,
> one could argue instead that every team should take a few hours a week
> to test a daily image, and I'd be all for that.  Of course, if we
> pick different teams per day, all the best bugs will get filed on
> Wednesday by the kernel team, because they're clearly the best non-QA
> image testers we've got. :P

I don't understand how the milestone freezes halt people in their tracks.  
There are a few people that are heavily involved in archive maintenance, but 
for most people I don't think landing things in the official archive a few days 
either way affects much.

> Seriously, though.  There's no way that a process that takes time can
> create time.  It's simple physics.  I understand the fear that dropping
> milestones could lead to less testing.  And, in a culture where we
> only test milestones, that's obviously true.  We need a cultural shift.

It's not creating time, it's shifting priorities.

> The only reason milestones sometimes still look like a "mess" and need
> time to settle is because we don't test enough between them.  If we
> tested enough between them, we wouldn't need milestones.  We've put
> serious (and I'd say quite successful) effort into making the archive
> installable and generally sane all cycle with the +1 maint effort, and
> I think it's time we extended that culture to a +1 QA culture, where
> testing is consistent, weird bugs are filed every day, and we don't
> have these 4-day long panic periods where WE HAVE TO FIX EVERY WEIRD
> BUG RIGHT NAO 'cause no one bothered to install from a daily for the
> last month.
> 
> Maybe I'm way off base.  Maybe I'm a nutter.  But I think I see where
> Rick's coming from here, and it's not about "muahaha, let's elminate
> testing by eliminating milestones", it's "hey, maybe we could be testing
> so consistently that milestones become an obvious waste of time".
> 
> ... Adam
> 
> PS: I leave you with a quote from IRC from the last milestone week:
>     "most of the bugs we know about, while not good, we can live with
>     for an A1, and they'll get fixed on the next daily."

Sure, but how did we find out about those bugs?  It was generally because we 
had the milestone and people tested. 

I think that at some point we may discover that our daily testing is sufficient 
that we don't need the pauses to do the milestones.  I think the way to do 
that is improve the daily testing and have multiple milestones with no 
surprises because we demonstrate the daily testing is adequate.  Any 
discussion of dropping things now is putting the cart way in front of the 
horse.

Scott K