Releasing Alphas and Betas without "freezing"

Fri Jun 15 19:21:13 UTC 2012

On 06/15/2012 11:19 AM, Stéphane Graber wrote:
> On 06/15/2012 10:46 AM, Scott Kitterman wrote:
>> On Friday, June 15, 2012 10:28:55 AM Stéphane Graber wrote:
>>> On 06/15/2012 10:12 AM, Rick Spencer wrote:
>>>> Hello all,
>>>>
>>>> At UDS I had some "hallway discussions" about why we freeze for Alphas
>>>> and Betas, and the fact that I think it is time to drop this practice
>>>> and rather focus on making Ubuntu good quality each day. Sadly, there
>>>> was no session on this, thus this email to this list for discussion.
>>>>
>>>> I think it is time drop our "Freeze" practices for the alphas and
>>>> betas. Here is my reasoning:
>>>>
>>>> 1. We are developing tools that allow us to efficiently use -proposed
>>>> in a way that will ensure we will not have partially built or
>>>> incompatible components in the release pocket ... ever. Including days
>>>> we release Alphas and Betas:
>>>>
>>>> These blueprints tools to ensure that Ubuntu is not uninstallable or
>>>> have other problems due to partially built components and such:
>>>> https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-interme
>>>> diary
>>>> https://blueprints.launchpad.net/ubuntu/+spec/other-q-freeze-use-of-propo
>>>> sed
>>>>
>>>> I have been assured that the tools necessary to automate the work of
>>>> moving components correctly from -proposed to the release will be
>>>> ready before Alpha 2.
>>>>
>>>> 2. We are investing heavily in the daily quality of Ubuntu. For example
>>>> ...
>>>> We run the same automated tests on an alpha as we run on a daily:
>>>> https://jenkins.qa.ubuntu.com/view/Quantal/view/ISO%20Testing%20Dashboard/
>>>>
>>>> We tend to archive issues each day:
>>>> http://people.canonical.com/~ubuntu-archive/testing/quantal_probs.html
>>>>
>>>> We ran all the manual ISO tests *before* we released Alpha 1, and we
>>>> have the capability of doing this at will:
>>>> http://iso.qa.ubuntu.com/qatracker/milestones/221/builds
>>>>
>>>> In short, freezing the archive before an alpha or beta should not
>>>> actually be contributing to either ensuring the installability of
>>>> Ubuntu images or ensuring the quality of these images. This implies,
>>>> therefore, that all the work around freezing, and all the productivity
>>>> lost during a freeze, actually subtracts from the quality of Ubuntu by
>>>> reducing our overall velocity for both features and bug fixes, since
>>>> every day the image is good quality, and Alpha or Beta should be just
>>>> that day's image tagged appropriately.
>>>>
>>>> AIUI, A1 was delivered in such a manner, though without the tooling to
>>>> ensure that moving from -proposed to the release pocket was efficient
>>>> and automated.
>>>>
>>>> Cheers, Rick
>>> Hi Rick,
>>>
>>> We certainly want to allow people to upload stuff to -proposed during a
>>> milestone week, but I don't agree that we should automatically copy from
>>> -proposed to the release pocket during a milestone week.
>>>
>>> We usually try to release all our images with the same versions of the
>>> packages, considering it takes us hours to rebuild everything, having
>>> seeded packages land during that time, will lead to images having
>>> different version of packages.
>>>
>>> As for what happened with Alpha 1, we simply asked everyone to upload
>>> their packages to -proposed and then cherry picked the packages we
>>> actually needed for the release from -proposed and copied them into the
>>> release pocket before rebuilding the images (we did that 3 times).
>>>
>>>
>>> As I understand it, the plan going forward is to have the release pocket
>>> be an alias of -proposed on upload, so that everything always lands into
>>> -proposed.
>>> After something lands in -proposed, is properly built and passes
>>> whatever other criteria we'll have, the package will be automatically
>>> copied to the release pocket.
>>>
>>> That last part (copy to the release pocket) would be what we'd block
>>> during a milestone week for any package that's seeded. These would be
>>> copied on a case by case basis by the release team and the images
>>> rebuilt afterwards.
>>>
>>> That'd essentially allow any non-seeded package to still flow to the
>>> release pocket and be available for everyone.
>>> All the others will be available for people running with -proposed
>>> enabled or will be available when we manually copy them to the release
>>> pocket or right after we release the milestone and we copy everything
>>> left in -proposed to the release pocket.
>> This looks complicated.  Maybe it will be easier in practice.
> Well, it'd be easier during hard freeze as things that aren't supposed
> to be frozen would still be automatically let through (but still
> preventing any archive skew).
>
>> It also looks like a big shift of work from developers to the release team.
>> Currently it's up to developers to decide what needs uploading to get to the
>> milestone.  During a soft freeze there's no action from the release team
>> except coordination.  With this mechanism, developers can upload whatever and
>> it's up to the release team to figure out.
> Yes, that's a bit of extra work, though it also gives us the guarantee
> that we won't have any accidental upload that might break the archive or
> require a respin.
>
> Packages that need to land in the release pocket for a milestone already
> need to be listed on the release team pad (as respin trigger), so I
> don't think it was much more work for alpha-1 to do the copy from
> -proposed for these. It certainly added a little delay (extra publisher
> cycle), but that's pretty much it.
>
>> I can see this being particularly a problem where a small fix is needed for the
>> milestone, but the developer's work would naturally flow to a larger update.
>> With no freeze and it being up to the release team, as Rick says, developer
>> flow would continue and the release team would get stuck without good choices
>> (I'm not sure if in this context there's a mechanism to do direct uploads to
>> the release pocket?).
> I don't have a good answer for that one, but direct upload to the
> release pocket would seem to be the easy way around that problem.
>
>> In Debian terms I'm seeing release-proposed as like unstable and release as
>> testing.  Is there a mechanism for direct uploads to testing (e.g. t-p-u)?
>>
>> Scott K
>
>
>
 From a community perspective, I would like to see us be able to 
concentrate our effort more towards making sure ubuntu is working 
properly throughout the cycle. Currently, we place focus on milestone 
testing for points of time, and we of course focus our testing on things 
deemed critical; for instance the installer gets quite a bit of focus at 
each milestone. I would rather see us having focused specific testing 
throughout the cycle, in addition to leveraging our community of people 
running the development version on a day to day basis.

So, given our "deliverable" at the end of the cycle is still an image, 
and our development teams are distributed and independent, I have a few 
thoughts:

What if we instead helped to enable our developers timelines for 
delivering new software in a quality manner? Things like being flexible 
on releasing, not freezing the archive, and enabling developers to do 
commit based unit testing

What if instead of several milestones, we had many small focused 
"releases" of software? I can imagine a managed timeline of deliveries 
with focused testing campaigns specific to the deliverables, and 
automated regression and integration testing happening to coincide.

How can we enable a better feedback loop for those who are running the 
development version of ubuntu everyday, and those who have a hand in 
producing it? Perhaps we could have the concept of a "beta" channel via 
the normal development archive, and turn proposed into "bleeding" edge 
with allowed breakage. Our metrics for quality then would be to deliver 
updates from proposed to the archive without rendering a users system to 
be un-usable for all essential hardware1.

Nicholas

1. Things that would render the system un-usable:
https://wiki.ubuntu.com/PrecisePangolin/DesktopCriticalPackages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20120615/bfb191b6/attachment-0001.html>