Releasing Alphas and Betas without "freezing"
mcasadevall at ubuntu.com
Thu Jun 21 18:28:15 UTC 2012
-----BEGIN PGP SIGNED MESSAGE-----
On 06/21/2012 11:20 AM, Scott Kitterman wrote:
> On Thursday, June 21, 2012 11:09:54 AM Michael Casadevall wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> On 06/21/2012 09:43 AM, Jono Bacon wrote:
>>> On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
>>> <mcasadevall at ubuntu.com> wrote:
>>>>>>> If we had more resources we would love to provide help
>>>>>>> for the flavors, and we are certainly happy to offer
>>>>>>> any guidance and advice, with with our current
>>>>>>> resources and staffing, Nick doesn't have the bandwidth
>>>>>>> to handle more the than the Ubuntu ISOs and associated
>>>>>>> testing. Saying that, I know Nick is in contact with
>>>>>>> many of the flavors to ensure they get the support they
>>>>>>> need to set up their own comprehensive testing plans.
>>>> The first I heard of any of this was when this email was sent
>>>> to u-devel. To be frank, changes of this magnitude should
>>>> have been discussed in seasons at UDS-Q, and not in hallway
>>>> conversations where many of the affected parties were not
>>> What changes of magnitude? We had already made it pretty clear
>>> that my team is not focused on flavors (historically we have
>>> never focused on flavors in our committed work items and always
>>> on Ubuntu), and all I am outlining here is the cadence in how
>>> Nick we be conducting his work. This is purely a way in which I
>>> choose to manage my resources to serve Ubuntu best.
>>> Yes, these plans were not discussed at UDS, because they have
>>> evolved since UDS, but Nick quite clearly laid out his testing
>>> strategy at UDS (which is not to dissimilar from this...he had
>>> scheduled testing plans for milestones and interim dailies).
>> It has already been established by Rick that we can increase
>> Canonical QA efforts without changing the milestone process. I'm
>> embedding Rick's proposal here:
>> On 06/17/2012 11:36 PM, Rick Spencer wrote:
>>> (I changed the subject to better represent this branch of the
>>> This discussion suggests that we don't need to release special
>>> alpha and beta ISOs, but we do need: 1. A cadence of testing 2.
>>> A trial run (or 2) of spinning ISOs 3. Development targets
>>> Therefore, I propose we: 1. Stop with the alphas and betas and
>>> win back all of the development effort 2. *Increase* the
>>> cadence of ISO testing to whatever we want or whatever the
>>> community team can manage 3. Spin a trial ISO near what is not
>>> beta time (maybe around current kernel freeze?) 4. Spin ISOs
>>> for release candidates 5. Maintain the current Alpha and Beta
>>> designations as development targets only (i.e. don't spin a
>>> special image for them).
>>> Cheers, Rick
>> The justification for this as I understand it is that the main
>> Ubuntu image is moving to more consistent and constant QA
>> practice, thus when combined with the active use of -proposed
>> during freezes, and other tools would render the freezes and
>> associates images moot.
>> We've already established that we can increase QA frequency
>> without changing the freeze/milestone process, and once the
>> -proposed queue is fully functional, no development effort will
>> be lost for those who do not wish to partake in testing; its
>> simply a matter then of enabling -proposed on ones machine
>>>> Many images such of Kubuntu have worked to have the various
>>>> milestones and deadlines set in ways that allow them to
>>>> incorperate the correct versions of their upstream packages
>>>> (i.e. KDE 4.9 release dates are more or less timed that the
>>>> RC and final will be available for Alpha 3 and Beta 1
>>>> Personal opinions aside, I object to large-scale changes in
>>>> release planning after everyone already agreed to the
>>>> current Quantal release schedule.
>>> The Kubuntu team are welcome to determine whatever milestones
>>> they want - no one is suggesting anything needs to change for
>>> the flavors in their testing cadence. I am purely stating how
>>> Nick will be working: the only output of this work is assured
>>> mandatory test case testing and this testing uncovering any
>>> bugs. As I said earlier, this work is independent of whether we
>>> remove milestones or not.
>> The Kubuntu team wants to have standard alpha and beta images
>> (i.e. the system that is in place and currently works for them).
>> As a member of the release team, and someone who has seen the
>> current system of freeze/milestone release catch image breaking
>> bugs, I want the official desktop/server images to be
>> cross-checked by the community by the same process that everyone
>> else uses.
>>>>> I don't think it is unreasonable for Canonical to focus
>>>>> its resources on Ubuntu as opposed to the flavors.
>>>> To what extent?
>>> To the extent that Canonical provides investment in Ubuntu, as
>>> part of this investment is paying Nick's salary, and as Nick's
>>> manager I choose how we resource his time and efforts. How we
>>> resource Nick is not a community decision.
>> I have no argument that Canonical directs Nick's work in whatever
>> way we see fit. However, as someone paid by Canonical to ensure
>> the high quality of our ARM images, I have a vested interest in
>> knowing if I'm covered these efforts.
>>>> As it stands, what is proposed will not only hinder but
>>>> likely cost considerable QA coverage of the many images that
>>>> are "community-supported". It is clear that non-Canonical
>>>> backed images simply can not support a rolling-QA testing
>>>> plan, and depend on the milestones system.
>>>> By changing how a minority of how images are being tested,
>>>> it will only serve to create confusion, and complications. As
>>>> a release manager, am I now to track down each team, make
>>>> sure from them that they've met whatever QA schedule they've
>>>> set for themselves, and then release off that?
>>>> As it stands, the vast majority of image testing comes at
>>>> milestones, and often picks up people who are otherwise
>>>> uninvolved in a flavors development. There has often been
>>>> calls to ask for additional testers for X image in
>>>> #ubuntu-testing should coverage be lacking, which have been
>>>> always answered.
>>> As I have said a few times now...If a flavor can't maintain
>>> the same level of testing, the flavor can just do the testing
>>> it can cope with. This might mean just doing the testing with
>>> the current cadence of milestones: just because I am ramping up
>>> our manual testing efforts with Nick does not mean anyone else
>>> has to change their own testing cadence.
>> As you stated later, ARM is not directly apart of these efforts
>> as of this moment. As such, the ARM images must remain on the
>> current freeze/release process.
>>> Flavors are in charge of their own destiny, and this includes
>>> testing cadence.
>> I believe the practical upshot of this is given that
>> community-flavors do not wish to abandon the system that is
>> simply easer to remove Ubuntu ISOs off the tracker (or otherwise
>> marked as non-essential/low priority) at milestone time and allow
>> everyone else to carry on as is.
>> Any community image that wishes to follow in Canonical's stead
>> is welcome to bring that up with the release manager, and after
>> making sure a proper process of QA reporting is ensured will then
>> be excused from the standard ISO tracking (and will not take part
>> of any image milestones except final).
>> To make sure that we don't release a sub-par product, all
>> flavors should undergo full testing at the end of the cycle, with
>> the understanding that should they not meet release standards,
>> they can and will be dropped.
>>>>> To be clear, the testing cadence is up to whatever the
>>>>> flavor wants it to be; I am not suggesting we impose this
>>>>> two-week cadence on anyone other than me imposing it on
>>>>> Nick. :-)
>>>> Then why change the existing system?
>>>> Nothing about the existing system will prevent the Canonical
>>>> QA efforts from implementing this.
>>> You are conflating two different things: testing cadence and
>>> milestones. To be clear: the cadence of Nick's testing has
>>> nothing to do with milestones and whether we choose to have
>>> them or not have them. My only point here is that I am
>>> disconnecting Nick's cadence from milestones so that if we do
>>> choose to not have them in the future, nothing changes.
>> No, I'm not. The former is being used to justify the removal of
>> the later, and the current system of alpha/betas.
>>>>> Naturally our flavors generally don't pay people to
>>>>> coordinate this kind of testing (maybe Blue Systems could
>>>>> invest here for Kubuntu?), but there is no requirement for
>>>>> the flavors to test every two weeks. If you folks want to
>>>>> test once a month or once every six weeks, then go ahead
>>>>> and do that. Importantly though, as with all flavors, you
>>>>> will need to coordinate this work yourselves within your
>>>> How does ARM factor into this?
>>>> I have already cited the failure by Canonical QA to test the
>>>> armhf+omap4 image for precise. Were it not for manual
>>>> intervention, and the normal Ubuntu QA testing by both myself
>>>> and other members of the community, I'm certain that we would
>>>> have shipped an image that was not only substandard, but
>>>> completely broken.
>>>> It should be cited for completeness that this was caught at
>>>> *Beta 2*, and only because there were voiced concerns on
>>>> image quality that I sat and tested it extensively.
>>>> Where were the QA efforts by Canonical QA for all the
>>>> previous milestones? As it stands, I'm only aware of one
>>>> person that was responsible for all ARM images that were
>>>> supported during precise.
>>>> Furthermore, speaking as the ARM Server Tech Lead in
>>>> Canonical, the PES team has to manually QA our own images
>>>> due to the fact that in general, we are the only teams with
>>>> access to the hardware. We do NOT have the bandwidth required
>>>> for this additional testing initiative.
>>> From my teams perspective, we were never asked to test ARM
>>> (and thus left it our of Nick's targeting), although this is
>>> something that we are going to be helping with moving forward.
>> Good to know.
>>>>> I see the milestones as quite disconnected from the
>>>>> testing: the milestones are a useful means of consolidating
>>>>> efforts around *something*, such as targeting work items
>>>>> and deadlines; it is just that we happen to use these
>>>>> milestones as a means to release. I personally don't see
>>>>> the point of the alpha releases...our users don't use them,
>>>>> and our developers are running the daily development branch
>>>>> anyway, so to me it makes sense to get rid of the
>>>>> milestones at some point.
>>>> I use milestones extremely regularly. Both to gauge our goals
>>>> by comparing what on the image is implemented to what is left
>>>> and a common place to gauge any bug fixes between milestones
>>>> to make sure we've fixed realistic issues between them. In
>>>> addition, it provides a time where a considerable part of the
>>>> developer community comes together to provide sanity checks
>>>> on images.
>>> As has been said before, no-one is suggesting milestones for
>>> targeting work are removed....
>> The confusion on the term milestone was already cleared up. Just
>> for the record, when I say milestone, I consider it milestone
>> images (mostly because that's when we confirm during QA testing
>> that everything slated for that milestone went in, and wasn't
>> accidently mismarked as completed or such).
>>>> In addition, as a rule (and excluding precise), during
>>>> development cycles, I generally only dist-upgrade my laptop
>>>> during freezes from milestone to milestone until after
>>>> Debian Import Freeze due to archive breakage and other
>>> ...and here lies the point: you should be able to upgrade
>>> *anytime*; not just at certain points. This is the gist behind
>>> Rick's points about assuring stability in our development
>> True, this reason for the milestones existing right now will be
>> resolved. That being said, it is only one part of the larger
>> picture, and many that do run the development releases upgrade on
>> a much more frequent timeframe.
>>>> Precise was considerably less broken during development due
>>>> to the simple fact we were syncing from Debian testing vs.
>>>> unstable, which ensured a minimum of package breaking due to
>>>> Debian's strict policies on how packages enter the testing
>>> That is one of the reasons why Precise was less broken, but
>>> there was also automated testing, acceptance criteria,
>>> increased manual testing, and a stronger focus on QA.
>> I'm awaiting an explanation on how a Canonical supported image
>> then was nearly released in a broken state at Beta 2.
>> One of the biggest hassles of running a development release is
>> packages going into an uninstallable state or otherwise skewing.
>> Syncing from testing solved this issue since our upstream for
>> the period was always in a consistent state and thus we did not
>> have issues such as transitions to worry about.
>> Once our equivalent of these tools are in place, then yes, we
>> will have a development release that can be upgraded both safely
>> and at will.
>>>>> My goal in this cycle is to ensure we have a regular
>>>>> testing cadence for Ubuntu and not based on milestones; if
>>>>> the Kubuntu team want to have your own internal milestones
>>>>> for targeting work and testing, I see no reason why you
>>>>> can't do that on your own schedule.
>>>> I can see a very good one. We are forsaking consistency
>>>> across all flavors for Canonical's interests. This even
>>>> affects the Ubuntu Desktop and Server images on
>>>> "community-supported" architectures as there is no one who
>>>> will be doing this work for armel or powerpc.
>>> This is not about "Canonicals' interests" this is about
>>> building efficiency in how we build Ubuntu; this efficiency
>>> benefits not just Canonical staff, but our wider community of
>>> participants who can assure a more stable development release.
>>> Nothing is being taken away here: flavors are still free to
>>> coordinate their testing as they see fit.
>> Scott has made it clear that if the current system of alpha/betas
>> is abolished, they will not be able to manage to maintain their
>> current level of quality. Kubuntu (and by extension, the other
>> flavors) depend on the current system as is.
>> As ARM and PowerPC is not actively part of these new QA efforts,
>> it is clear that we are also dependent on the current system to
>> keep quality.
>> It should be noted though that as long as one architecture in an
>> image can't proscribe to the new testing method, the packageset
>> relating to that image in the release pocket must be frozen as we
>> have no way of freezing on a per-architecture basis.
> If Ubuntu wants to skip the Alpha/Beta milestones because Canonical
> thinks it's other QA efforts are sufficient, I think they should
> feel free to do it, but the existing structure needs to stay in
> place for the rest of us.
> Scott K
I think the basic process should now be something like this:
1. Images can opt-out of the existing QA mechanisms once they
understand the full risks involved of dong so.
2. Images that have out-opted are free to ignore freezes for their
respective package sets (feature freeze and such still apply of course).
As a thought, while the packageset in release must be frozen if one of
the architectures used by that image do not opt-out, we can simply
enabled proposed on the dailies to allow unbroken development efforts.
3. For pre-release freezes, all images will be re-added to the QA
tracker, and undergo normal QA processes
4. As per normal, any image that fails will be excluded from the
release (or 12.10 in this case).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
-----END PGP SIGNATURE-----
More information about the ubuntu-devel