Releasing Alphas and Betas without "freezing"

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/21/2012 09:43 AM, Jono Bacon wrote:
> On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall 
> <mcasadevall at ubuntu.com> wrote:
>>>>> If we had more resources we would love to provide help for 
>>>>> the flavors, and we are certainly happy to offer any 
>>>>> guidance and advice, with with our current resources and 
>>>>> staffing, Nick doesn't have the bandwidth to handle more 
>>>>> the than the Ubuntu ISOs and associated testing. Saying 
>>>>> that, I know Nick is in contact with many of the flavors to
>>>>> ensure they get the support they need to set up their own
>>>>> comprehensive testing plans.
>>>> 
>> 
>> The first I heard of any of this was when this email was sent to 
>> u-devel. To be frank, changes of this magnitude should have been 
>> discussed in seasons at UDS-Q, and not in hallway conversations 
>> where many of the affected parties were not present.
> 
> What changes of magnitude? We had already made it pretty clear that
> my team is not focused on flavors (historically we have never 
> focused on flavors in our committed work items and always on 
> Ubuntu), and all I am outlining here is the cadence in how Nick we 
> be conducting his work. This is purely a way in which I choose to 
> manage my resources to serve Ubuntu best.
> 
> Yes, these plans were not discussed at UDS, because they have 
> evolved since UDS, but Nick quite clearly laid out his testing 
> strategy at UDS (which is not to dissimilar from this...he had 
> scheduled testing plans for milestones and interim dailies).
> 

It has already been established by Rick that we can increase Canonical
QA efforts without changing the milestone process. I'm embedding
Rick's proposal here:

On 06/17/2012 11:36 PM, Rick Spencer wrote:
> (I changed the subject to better represent this branch of the
conversation)
> 
> This discussion suggests that we don't need to release special 
> alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A 
> trial run (or 2) of spinning ISOs 3. Development targets
> 
> Therefore, I propose we: 1. Stop with the alphas and betas and win 
> back all of the development effort 2. *Increase* the cadence of ISO
> testing to whatever we want or whatever the community team can
> manage 3. Spin a trial ISO near what is not beta time (maybe around
> current kernel freeze?) 4. Spin ISOs for release candidates 5.
> Maintain the current Alpha and Beta designations as development
> targets only (i.e. don't spin a special image for them).
> 
> Cheers, Rick

The justification for this as I understand it is that the main Ubuntu
image is moving to more consistent and constant QA practice, thus when
combined with the active use of -proposed during freezes, and other
tools would render the freezes and associates images moot.

We've already established that we can increase QA frequency without
changing the freeze/milestone process, and once the -proposed queue is
fully functional, no development effort will be lost for those who do
not wish to partake in testing; its simply a matter then of enabling
 -proposed on ones machine

>> Many images such of Kubuntu have worked to have the various 
>> milestones and deadlines set in ways that allow them to 
>> incorperate the correct versions of their upstream packages (i.e.
>> KDE 4.9 release dates are more or less timed that the RC and
>> final will be available for Alpha 3 and Beta 1 respectively).
>> 
>> Personal opinions aside, I object to large-scale changes in 
>> release planning after everyone already agreed to the current 
>> Quantal release schedule.
> 
> The Kubuntu team are welcome to determine whatever milestones they 
> want - no one is suggesting anything needs to change for the 
> flavors in their testing cadence. I am purely stating how Nick will
> be working: the only output of this work is assured mandatory test
> case testing and this testing uncovering any bugs. As I said 
> earlier, this work is independent of whether we remove milestones 
> or not.
> 

The Kubuntu team wants to have standard alpha and beta images (i.e.
the system that is in place and currently works for them).

As a member of the release team, and someone who has seen the current
system of freeze/milestone release catch image breaking bugs, I want
the official desktop/server images to be cross-checked by the
community by the same process that everyone else uses.

>>> I don't think it is unreasonable for Canonical to focus its 
>>> resources on Ubuntu as opposed to the flavors.
>>> 
>> 
>> To what extent?
> 
> To the extent that Canonical provides investment in Ubuntu, as part
> of this investment is paying Nick's salary, and as Nick's manager I
> choose how we resource his time and efforts. How we resource Nick
> is not a community decision.
> 

I have no argument that Canonical directs Nick's work in whatever way
we see fit. However, as someone paid by Canonical to ensure the high
quality of our ARM images, I have a vested interest in knowing if I'm
covered these efforts.

>> As it stands, what is proposed will not only hinder but likely 
>> cost considerable QA coverage of the many images that are 
>> "community-supported". It is clear that non-Canonical backed 
>> images simply can not support a rolling-QA testing plan, and 
>> depend on the milestones system.
>> 
>> By changing how a minority of how images are being tested, it 
>> will only serve to create confusion, and complications. As a 
>> release manager, am I now to track down each team, make sure from
>> them that they've met whatever QA schedule they've set for 
>> themselves, and then release off that?
>> 
>> As it stands, the vast majority of image testing comes at 
>> milestones, and often picks up people who are otherwise 
>> uninvolved in a flavors development. There has often been calls 
>> to ask for additional testers for X image in #ubuntu-testing 
>> should coverage be lacking, which have been always answered.
> 
> As I have said a few times now...If a flavor can't maintain the 
> same level of testing, the flavor can just do the testing it can 
> cope with. This might mean just doing the testing with the current 
> cadence of milestones: just because I am ramping up our manual 
> testing efforts with Nick does not mean anyone else has to change 
> their own testing cadence.
> 

As you stated later, ARM is not directly apart of these efforts as of
this moment. As such, the ARM images must remain on the current
freeze/release process.

> Flavors are in charge of their own destiny, and this includes 
> testing cadence.
> 

I believe the practical upshot of this is given that community-flavors
do not wish to abandon the system that is simply easer to remove
Ubuntu ISOs off the tracker (or otherwise marked as non-essential/low
priority) at milestone time and allow everyone else to carry on as is.

Any community image that wishes to follow in Canonical's stead is
welcome to bring that up with the release manager, and after making
sure a proper process of QA reporting is ensured will then be excused
from the standard ISO tracking (and will not take part of any image
milestones except final).

To make sure that we don't release a sub-par product, all flavors
should undergo full testing at the end of the cycle, with the
understanding that should they not meet release standards, they can
and will be dropped.

>>> To be clear, the testing cadence is up to whatever the flavor 
>>> wants it to be; I am not suggesting we impose this two-week 
>>> cadence on anyone other than me imposing it on Nick. :-)
>>> 
>> 
>> Then why change the existing system?
>> 
>> Nothing about the existing system will prevent the Canonical QA 
>> efforts from implementing this.
> 
> You are conflating two different things: testing cadence and 
> milestones. To be clear: the cadence of Nick's testing has nothing 
> to do with milestones and whether we choose to have them or not 
> have them. My only point here is that I am disconnecting Nick's 
> cadence from milestones so that if we do choose to not have them in
> the future, nothing changes.
> 

No, I'm not. The former is being used to justify the removal of the
later, and the current system of alpha/betas.

>>> Naturally our flavors generally don't pay people to coordinate 
>>> this kind of testing (maybe Blue Systems could invest here for 
>>> Kubuntu?), but there is no requirement for the flavors to test 
>>> every two weeks. If you folks want to test once a month or once
>>> every six weeks, then go ahead and do that. Importantly though,
>>> as with all flavors, you will need to coordinate this work
>>> yourselves within your community.
>>> 
>> 
>> How does ARM factor into this?
>> 
>> I have already cited the failure by Canonical QA to test the 
>> armhf+omap4 image for precise. Were it not for manual 
>> intervention, and the normal Ubuntu QA testing by both myself and
>> other members of the community, I'm certain that we would have
>> shipped an image that was not only substandard, but completely
>> broken.
>> 
>> It should be cited for completeness that this was caught at *Beta
>> 2*, and only because there were voiced concerns on image quality
>> that I sat and tested it extensively.
>> 
>> Where were the QA efforts by Canonical QA for all the previous 
>> milestones? As it stands, I'm only aware of one person that was 
>> responsible for all ARM images that were supported during 
>> precise.
>> 
>> Furthermore, speaking as the ARM Server Tech Lead in Canonical, 
>> the PES team has to  manually QA our own images due to the fact 
>> that in general, we are the only teams with access to the 
>> hardware. We do NOT have the bandwidth required for this 
>> additional testing initiative.
> 
> From my teams perspective, we were never asked to test ARM (and 
> thus left it our of Nick's targeting), although this is something 
> that we are going to be helping with moving forward.
> 

Good to know.

>>> I see the milestones as quite disconnected from the testing: 
>>> the milestones are a useful means of consolidating efforts 
>>> around *something*, such as targeting work items and deadlines;
>>> it is just that we happen to use these milestones as a means to
>>> release. I personally don't see the point of the alpha
>>> releases...our users don't use them, and our developers are
>>> running the daily development branch anyway, so to me it makes
>>> sense to get rid of the milestones at some point.
>>> 
>> 
>> I use milestones extremely regularly. Both to gauge our goals by 
>> comparing what on the image is implemented to what is left and a 
>> common place to gauge any bug fixes between milestones to make 
>> sure we've fixed realistic issues between them. In addition, it 
>> provides a time where a considerable part of the developer 
>> community comes together to provide sanity checks on images.
> 
> As has been said before, no-one is suggesting milestones for 
> targeting work are removed....
> 

The confusion on the term milestone was already cleared up. Just for
the record, when I say milestone, I consider it milestone images
(mostly because that's when we confirm during QA testing that
everything slated for that milestone went in, and wasn't accidently
mismarked as completed or such).

>> In addition, as a rule (and excluding precise), during 
>> development cycles, I generally only dist-upgrade my laptop 
>> during freezes from milestone to milestone until after Debian 
>> Import Freeze due to archive breakage and other issues.
> 
> ...and here lies the point: you should be able to upgrade 
> *anytime*; not just at certain points. This is the gist behind 
> Rick's points about assuring stability in our development release.
> 

True, this reason for the milestones existing right now will be
resolved. That being said, it is only one part of the larger picture,
and many that do run the development releases upgrade on a much more
frequent timeframe.

>> Precise was considerably less broken during development due to 
>> the simple fact we were syncing from Debian testing vs. unstable,
>> which ensured a minimum of package breaking due to Debian's
>> strict policies on how packages enter the testing archive.
> 
> That is one of the reasons why Precise was less broken, but there 
> was also automated testing, acceptance criteria, increased manual 
> testing, and a stronger focus on QA.
> 

I'm awaiting an explanation on how a Canonical supported image then
was nearly released in a broken state at Beta 2.

One of the biggest hassles of running a development release is
packages going into an uninstallable state or otherwise skewing.
Syncing from testing solved this issue since our upstream for the
period was always in a consistent state and thus we did not have
issues such as transitions to worry about.

Once our equivalent of these tools are in place, then yes, we will
have a development release that can be upgraded both safely and at will.

>>> My goal in this cycle is to ensure we have a regular testing 
>>> cadence for Ubuntu and not based on milestones; if the Kubuntu 
>>> team want to have your own internal milestones for targeting 
>>> work and testing, I see no reason why you can't do that on your
>>> own schedule.
>>> 
>> 
>> I can see a very good one. We are forsaking consistency across 
>> all flavors for Canonical's interests. This even affects the 
>> Ubuntu Desktop and Server images on "community-supported" 
>> architectures as there is no one who will be doing this work for 
>> armel or powerpc.
> 
> This is not about "Canonicals' interests" this is about building 
> efficiency in how we build Ubuntu; this efficiency benefits not 
> just Canonical staff, but our wider community of participants who 
> can assure a more stable development release.
> 
> Nothing is being taken away here: flavors are still free to 
> coordinate their testing as they see fit.
> 
> Jono
> 

Scott has made it clear that if the current system of alpha/betas is
abolished, they will not be able to manage to maintain their current
level of quality. Kubuntu (and by extension, the other flavors) depend
on the current system as is.

As ARM and PowerPC is not actively part of these new QA efforts, it is
clear that we are also dependent on the current system to keep quality.

It should be noted though that as long as one architecture in an image
can't proscribe to the new testing method, the packageset relating to
that image in the release pocket must be frozen as we have no way of
freezing on a per-architecture basis.


Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP42NxAAoJEHM+GkLSJHY5O+8P/3u2LIwVLtP2ybxBxLcL9viv
qJhxMezclZerpNAB55sjrOYTr6NdN8J6bBkHOPFlXOkdSeRB9D1CSbsvWDm8CGFJ
a5lpgLVWBz4qaC5/YHB7vgcV/Huv91lWd96h0i48gmWbF3nsIbwgJnl8+gs2F0g+
cDLIY8Q01eZJTDjE0XVZr0Cnhwm6bGgFOIT4Sbi9T1dXyqdi8q79aq3EuaA7LNhY
XxayVssuowwTHIWRrtS18uYYcM/GH9jKDFX9aNof+3F9tOWA7uktNqEqhYohdxY6
9hGu6ZoMWKayX50oQ/RQT6Al2jaxamPgjvKStLnXmexsGkALJGKl++wG5uC5hdhk
qrhrBud+ExHerhvFVNPbu0y936hph58ha2OeabVNt7LtimUQYqdXFLaVN+rEJ/V/
8tSaDbq+1x5kzlq51gHHjPLjKav7ALrSp2ZT9SdTQo91D0lvwMdD8uXz9pwC4d3p
LaAr3uGOFMyZiawGfpFWs724lcS84hC1ia+BQjWjzbV+PSyJtyEFjJqgawTF07Nn
Qa+E5yYsciE4WGTijxevsVJykRpojVx+bcRtGKCZjCu9j/ZOkClCOGlWG5E+rBRe
7fkPESOXkOKJveNnc/9XceUOkvpb2ijU4j9iPV2D7HxyVSgOuh2Q+PmKiDYOSktY
0HKERvgwqMn4O3OiQBcT
=vuaU
-----END PGP SIGNATURE-----