Releasing Alphas and Betas without "freezing"

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/21/2012 11:20 AM, Scott Kitterman wrote:
> On Thursday, June 21, 2012 11:09:54 AM Michael Casadevall wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> 
>> On 06/21/2012 09:43 AM, Jono Bacon wrote:
>>> On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
>>> 
>>> <mcasadevall at ubuntu.com> wrote:
>>>>>>> If we had more resources we would love to provide help
>>>>>>> for the flavors, and we are certainly happy to offer
>>>>>>> any guidance and advice, with with our current
>>>>>>> resources and staffing, Nick doesn't have the bandwidth
>>>>>>> to handle more the than the Ubuntu ISOs and associated
>>>>>>> testing. Saying that, I know Nick is in contact with
>>>>>>> many of the flavors to ensure they get the support they
>>>>>>> need to set up their own comprehensive testing plans.
>>>> 
>>>> The first I heard of any of this was when this email was sent
>>>> to u-devel. To be frank, changes of this magnitude should
>>>> have been discussed in seasons at UDS-Q, and not in hallway
>>>> conversations where many of the affected parties were not
>>>> present.
>>> 
>>> What changes of magnitude? We had already made it pretty clear
>>> that my team is not focused on flavors (historically we have
>>> never focused on flavors in our committed work items and always
>>> on Ubuntu), and all I am outlining here is the cadence in how
>>> Nick we be conducting his work. This is purely a way in which I
>>> choose to manage my resources to serve Ubuntu best.
>>> 
>>> Yes, these plans were not discussed at UDS, because they have 
>>> evolved since UDS, but Nick quite clearly laid out his testing 
>>> strategy at UDS (which is not to dissimilar from this...he had 
>>> scheduled testing plans for milestones and interim dailies).
>> 
>> It has already been established by Rick that we can increase
>> Canonical QA efforts without changing the milestone process. I'm
>> embedding Rick's proposal here:
>> 
>> On 06/17/2012 11:36 PM, Rick Spencer wrote:
>>> (I changed the subject to better represent this branch of the
>> 
>> conversation)
>> 
>>> This discussion suggests that we don't need to release special 
>>> alpha and beta ISOs, but we do need: 1. A cadence of testing 2.
>>> A trial run (or 2) of spinning ISOs 3. Development targets
>>> 
>>> Therefore, I propose we: 1. Stop with the alphas and betas and
>>> win back all of the development effort 2. *Increase* the
>>> cadence of ISO testing to whatever we want or whatever the
>>> community team can manage 3. Spin a trial ISO near what is not
>>> beta time (maybe around current kernel freeze?) 4. Spin ISOs
>>> for release candidates 5. Maintain the current Alpha and Beta
>>> designations as development targets only (i.e. don't spin a
>>> special image for them).
>>> 
>>> Cheers, Rick
>> 
>> The justification for this as I understand it is that the main
>> Ubuntu image is moving to more consistent and constant QA
>> practice, thus when combined with the active use of -proposed
>> during freezes, and other tools would render the freezes and
>> associates images moot.
>> 
>> We've already established that we can increase QA frequency
>> without changing the freeze/milestone process, and once the
>> -proposed queue is fully functional, no development effort will
>> be lost for those who do not wish to partake in testing; its
>> simply a matter then of enabling -proposed on ones machine
>> 
>>>> Many images such of Kubuntu have worked to have the various 
>>>> milestones and deadlines set in ways that allow them to 
>>>> incorperate the correct versions of their upstream packages
>>>> (i.e. KDE 4.9 release dates are more or less timed that the
>>>> RC and final will be available for Alpha 3 and Beta 1
>>>> respectively).
>>>> 
>>>> Personal opinions aside, I object to large-scale changes in 
>>>> release planning after everyone already agreed to the
>>>> current Quantal release schedule.
>>> 
>>> The Kubuntu team are welcome to determine whatever milestones
>>> they want - no one is suggesting anything needs to change for
>>> the flavors in their testing cadence. I am purely stating how
>>> Nick will be working: the only output of this work is assured
>>> mandatory test case testing and this testing uncovering any
>>> bugs. As I said earlier, this work is independent of whether we
>>> remove milestones or not.
>> 
>> The Kubuntu team wants to have standard alpha and beta images
>> (i.e. the system that is in place and currently works for them).
>> 
>> As a member of the release team, and someone who has seen the
>> current system of freeze/milestone release catch image breaking
>> bugs, I want the official desktop/server images to be
>> cross-checked by the community by the same process that everyone
>> else uses.
>> 
>>>>> I don't think it is unreasonable for Canonical to focus
>>>>> its resources on Ubuntu as opposed to the flavors.
>>>> 
>>>> To what extent?
>>> 
>>> To the extent that Canonical provides investment in Ubuntu, as
>>> part of this investment is paying Nick's salary, and as Nick's
>>> manager I choose how we resource his time and efforts. How we
>>> resource Nick is not a community decision.
>> 
>> I have no argument that Canonical directs Nick's work in whatever
>> way we see fit. However, as someone paid by Canonical to ensure
>> the high quality of our ARM images, I have a vested interest in
>> knowing if I'm covered these efforts.
>> 
>>>> As it stands, what is proposed will not only hinder but
>>>> likely cost considerable QA coverage of the many images that
>>>> are "community-supported". It is clear that non-Canonical
>>>> backed images simply can not support a rolling-QA testing
>>>> plan, and depend on the milestones system.
>>>> 
>>>> By changing how a minority of how images are being tested,
>>>> it will only serve to create confusion, and complications. As
>>>> a release manager, am I now to track down each team, make
>>>> sure from them that they've met whatever QA schedule they've
>>>> set for themselves, and then release off that?
>>>> 
>>>> As it stands, the vast majority of image testing comes at 
>>>> milestones, and often picks up people who are otherwise 
>>>> uninvolved in a flavors development. There has often been
>>>> calls to ask for additional testers for X image in
>>>> #ubuntu-testing should coverage be lacking, which have been
>>>> always answered.
>>> 
>>> As I have said a few times now...If a flavor can't maintain
>>> the same level of testing, the flavor can just do the testing
>>> it can cope with. This might mean just doing the testing with
>>> the current cadence of milestones: just because I am ramping up
>>> our manual testing efforts with Nick does not mean anyone else
>>> has to change their own testing cadence.
>> 
>> As you stated later, ARM is not directly apart of these efforts
>> as of this moment. As such, the ARM images must remain on the
>> current freeze/release process.
>> 
>>> Flavors are in charge of their own destiny, and this includes 
>>> testing cadence.
>> 
>> I believe the practical upshot of this is given that
>> community-flavors do not wish to abandon the system that is
>> simply easer to remove Ubuntu ISOs off the tracker (or otherwise
>> marked as non-essential/low priority) at milestone time and allow
>> everyone else to carry on as is.
>> 
>> Any community image that wishes to follow in Canonical's stead
>> is welcome to bring that up with the release manager, and after
>> making sure a proper process of QA reporting is ensured will then
>> be excused from the standard ISO tracking (and will not take part
>> of any image milestones except final).
>> 
>> To make sure that we don't release a sub-par product, all
>> flavors should undergo full testing at the end of the cycle, with
>> the understanding that should they not meet release standards,
>> they can and will be dropped.
>> 
>>>>> To be clear, the testing cadence is up to whatever the
>>>>> flavor wants it to be; I am not suggesting we impose this
>>>>> two-week cadence on anyone other than me imposing it on
>>>>> Nick. :-)
>>>> 
>>>> Then why change the existing system?
>>>> 
>>>> Nothing about the existing system will prevent the Canonical
>>>> QA efforts from implementing this.
>>> 
>>> You are conflating two different things: testing cadence and 
>>> milestones. To be clear: the cadence of Nick's testing has
>>> nothing to do with milestones and whether we choose to have
>>> them or not have them. My only point here is that I am
>>> disconnecting Nick's cadence from milestones so that if we do
>>> choose to not have them in the future, nothing changes.
>> 
>> No, I'm not. The former is being used to justify the removal of
>> the later, and the current system of alpha/betas.
>> 
>>>>> Naturally our flavors generally don't pay people to
>>>>> coordinate this kind of testing (maybe Blue Systems could
>>>>> invest here for Kubuntu?), but there is no requirement for
>>>>> the flavors to test every two weeks. If you folks want to
>>>>> test once a month or once every six weeks, then go ahead
>>>>> and do that. Importantly though, as with all flavors, you
>>>>> will need to coordinate this work yourselves within your
>>>>> community.
>>>> 
>>>> How does ARM factor into this?
>>>> 
>>>> I have already cited the failure by Canonical QA to test the 
>>>> armhf+omap4 image for precise. Were it not for manual 
>>>> intervention, and the normal Ubuntu QA testing by both myself
>>>> and other members of the community, I'm certain that we would
>>>> have shipped an image that was not only substandard, but
>>>> completely broken.
>>>> 
>>>> It should be cited for completeness that this was caught at
>>>> *Beta 2*, and only because there were voiced concerns on
>>>> image quality that I sat and tested it extensively.
>>>> 
>>>> Where were the QA efforts by Canonical QA for all the
>>>> previous milestones? As it stands, I'm only aware of one
>>>> person that was responsible for all ARM images that were
>>>> supported during precise.
>>>> 
>>>> Furthermore, speaking as the ARM Server Tech Lead in
>>>> Canonical, the PES team has to  manually QA our own images
>>>> due to the fact that in general, we are the only teams with
>>>> access to the hardware. We do NOT have the bandwidth required
>>>> for this additional testing initiative.
>>> 
>>> From my teams perspective, we were never asked to test ARM
>>> (and thus left it our of Nick's targeting), although this is
>>> something that we are going to be helping with moving forward.
>> 
>> Good to know.
>> 
>>>>> I see the milestones as quite disconnected from the
>>>>> testing: the milestones are a useful means of consolidating
>>>>> efforts around *something*, such as targeting work items
>>>>> and deadlines; it is just that we happen to use these
>>>>> milestones as a means to release. I personally don't see
>>>>> the point of the alpha releases...our users don't use them,
>>>>> and our developers are running the daily development branch
>>>>> anyway, so to me it makes sense to get rid of the
>>>>> milestones at some point.
>>>> 
>>>> I use milestones extremely regularly. Both to gauge our goals
>>>> by comparing what on the image is implemented to what is left
>>>> and a common place to gauge any bug fixes between milestones
>>>> to make sure we've fixed realistic issues between them. In
>>>> addition, it provides a time where a considerable part of the
>>>> developer community comes together to provide sanity checks
>>>> on images.
>>> 
>>> As has been said before, no-one is suggesting milestones for 
>>> targeting work are removed....
>> 
>> The confusion on the term milestone was already cleared up. Just
>> for the record, when I say milestone, I consider it milestone
>> images (mostly because that's when we confirm during QA testing
>> that everything slated for that milestone went in, and wasn't
>> accidently mismarked as completed or such).
>> 
>>>> In addition, as a rule (and excluding precise), during 
>>>> development cycles, I generally only dist-upgrade my laptop 
>>>> during freezes from milestone to milestone until after
>>>> Debian Import Freeze due to archive breakage and other
>>>> issues.
>>> 
>>> ...and here lies the point: you should be able to upgrade 
>>> *anytime*; not just at certain points. This is the gist behind 
>>> Rick's points about assuring stability in our development
>>> release.
>> 
>> True, this reason for the milestones existing right now will be 
>> resolved. That being said, it is only one part of the larger
>> picture, and many that do run the development releases upgrade on
>> a much more frequent timeframe.
>> 
>>>> Precise was considerably less broken during development due
>>>> to the simple fact we were syncing from Debian testing vs.
>>>> unstable, which ensured a minimum of package breaking due to
>>>> Debian's strict policies on how packages enter the testing
>>>> archive.
>>> 
>>> That is one of the reasons why Precise was less broken, but
>>> there was also automated testing, acceptance criteria,
>>> increased manual testing, and a stronger focus on QA.
>> 
>> I'm awaiting an explanation on how a Canonical supported image
>> then was nearly released in a broken state at Beta 2.
>> 
>> One of the biggest hassles of running a development release is 
>> packages going into an uninstallable state or otherwise skewing. 
>> Syncing from testing solved this issue since our upstream for
>> the period was always in a consistent state and thus we did not
>> have issues such as transitions to worry about.
>> 
>> Once our equivalent of these tools are in place, then yes, we
>> will have a development release that can be upgraded both safely
>> and at will.
>> 
>>>>> My goal in this cycle is to ensure we have a regular
>>>>> testing cadence for Ubuntu and not based on milestones; if
>>>>> the Kubuntu team want to have your own internal milestones
>>>>> for targeting work and testing, I see no reason why you
>>>>> can't do that on your own schedule.
>>>> 
>>>> I can see a very good one. We are forsaking consistency
>>>> across all flavors for Canonical's interests. This even
>>>> affects the Ubuntu Desktop and Server images on
>>>> "community-supported" architectures as there is no one who
>>>> will be doing this work for armel or powerpc.
>>> 
>>> This is not about "Canonicals' interests" this is about
>>> building efficiency in how we build Ubuntu; this efficiency
>>> benefits not just Canonical staff, but our wider community of
>>> participants who can assure a more stable development release.
>>> 
>>> Nothing is being taken away here: flavors are still free to 
>>> coordinate their testing as they see fit.
>>> 
>>> Jono
>> 
>> Scott has made it clear that if the current system of alpha/betas
>> is abolished, they will not be able to manage to maintain their
>> current level of quality. Kubuntu (and by extension, the other
>> flavors) depend on the current system as is.
>> 
>> As ARM and PowerPC is not actively part of these new QA efforts,
>> it is clear that we are also dependent on the current system to
>> keep quality.
>> 
>> It should be noted though that as long as one architecture in an
>> image can't proscribe to the new testing method, the packageset
>> relating to that image in the release pocket must be frozen as we
>> have no way of freezing on a per-architecture basis.
> 
> +1.
> 
> If Ubuntu wants to skip the Alpha/Beta milestones because Canonical
> thinks it's other QA efforts are sufficient, I think they should
> feel free to do it, but the existing structure needs to stay in
> place for the rest of us.
> 
> Scott K
> 

I think the basic process should now be something like this:

1. Images can opt-out of the existing QA mechanisms once they
understand the full risks involved of dong so.

2. Images that have out-opted are free to ignore freezes for their
respective package sets (feature freeze and such still apply of course).

As a thought, while the packageset in release must be frozen if one of
the architectures used by that image do not opt-out, we can simply
enabled proposed on the dailies to allow unbroken development efforts.

3. For pre-release freezes, all images will be re-added to the QA
tracker, and undergo normal QA processes

4. As per normal, any image that fails will be excluded from the
release (or 12.10 in this case).

Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP42e+AAoJEHM+GkLSJHY5pkIQAJFYmxbA4rd7NYI6YIi7M5DR
9Moj+N4Mnp6LIwPdZa6lqXNqqgTAkrMZ0fp4TNrl1RaWEFz9/hExa2Lmq/5T8SHe
1naz/1Un3mxQhI1i6xoLdKqVrjy3JCCa5u6el/S769t6J1qgwsOeOQMBQnpkOzzR
LpTw48CiS0tH5HWCJQC535+bquU3md77ttdkhsdz/zoYm+MgW7/q+LjbiFvglvEb
gE8jxnJSvdR4U79bjw+otrE8wHMMvKezGJ5KIYxI3n5KuHWkOioOq/pouXdkUiXh
zpIw7GT5n/opQNH0Z7vm1Iagj255Wbkiv2/+AM31wc4m+cD5WHZZmY+fyP0PTCKT
i02wK3s0PI3fAOrLaRrYT/VY9vxRoW+/WSuNAkYjPsPywVEj0dqQOfa0UPZ1EQXM
xCI2q6RlJky5i1EFoyEurl1TIcm+fdHJY+mqfEAnBZ1HAZwo8I4qguZ6NmwziWz3
HZTRcccSZ5uL3WoyYZ8PXzAAimO7hHjnQRHE9b5hXJjd4L98RE2sFtj4repCct25
MExgzhhUBY3J8q8vZCGADY0Z39F+/0es4gyzzop4qVOTKRFWHOcqwfkO551I8oK5
8xRE8M7sOuTsCg6ncfD0uAOFplIcnKGjHmhfCDZqP2LKJuujPYLeAzZY1hlg4cpQ
Co/Fo9suN3KKsES+WGXl
=7ubS
-----END PGP SIGNATURE-----