Releasing Alphas and Betas without "freezing"

[Scott Kitterman]

On Thursday, June 21, 2012 11:09:54 AM Michael Casadevall wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 06/21/2012 09:43 AM, Jono Bacon wrote:
> > On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
> > 
> > <mcasadevall at ubuntu.com> wrote:
> >>>>> If we had more resources we would love to provide help for
> >>>>> the flavors, and we are certainly happy to offer any
> >>>>> guidance and advice, with with our current resources and
> >>>>> staffing, Nick doesn't have the bandwidth to handle more
> >>>>> the than the Ubuntu ISOs and associated testing. Saying
> >>>>> that, I know Nick is in contact with many of the flavors to
> >>>>> ensure they get the support they need to set up their own
> >>>>> comprehensive testing plans.
> >> 
> >> The first I heard of any of this was when this email was sent to
> >> u-devel. To be frank, changes of this magnitude should have been
> >> discussed in seasons at UDS-Q, and not in hallway conversations
> >> where many of the affected parties were not present.
> > 
> > What changes of magnitude? We had already made it pretty clear that
> > my team is not focused on flavors (historically we have never
> > focused on flavors in our committed work items and always on
> > Ubuntu), and all I am outlining here is the cadence in how Nick we
> > be conducting his work. This is purely a way in which I choose to
> > manage my resources to serve Ubuntu best.
> > 
> > Yes, these plans were not discussed at UDS, because they have
> > evolved since UDS, but Nick quite clearly laid out his testing
> > strategy at UDS (which is not to dissimilar from this...he had
> > scheduled testing plans for milestones and interim dailies).
> 
> It has already been established by Rick that we can increase Canonical
> QA efforts without changing the milestone process. I'm embedding
> Rick's proposal here:
> 
> On 06/17/2012 11:36 PM, Rick Spencer wrote:
> > (I changed the subject to better represent this branch of the
> 
> conversation)
> 
> > This discussion suggests that we don't need to release special
> > alpha and beta ISOs, but we do need: 1. A cadence of testing 2. A
> > trial run (or 2) of spinning ISOs 3. Development targets
> > 
> > Therefore, I propose we: 1. Stop with the alphas and betas and win
> > back all of the development effort 2. *Increase* the cadence of ISO
> > testing to whatever we want or whatever the community team can
> > manage 3. Spin a trial ISO near what is not beta time (maybe around
> > current kernel freeze?) 4. Spin ISOs for release candidates 5.
> > Maintain the current Alpha and Beta designations as development
> > targets only (i.e. don't spin a special image for them).
> > 
> > Cheers, Rick
> 
> The justification for this as I understand it is that the main Ubuntu
> image is moving to more consistent and constant QA practice, thus when
> combined with the active use of -proposed during freezes, and other
> tools would render the freezes and associates images moot.
> 
> We've already established that we can increase QA frequency without
> changing the freeze/milestone process, and once the -proposed queue is
> fully functional, no development effort will be lost for those who do
> not wish to partake in testing; its simply a matter then of enabling
>  -proposed on ones machine
> 
> >> Many images such of Kubuntu have worked to have the various
> >> milestones and deadlines set in ways that allow them to
> >> incorperate the correct versions of their upstream packages (i.e.
> >> KDE 4.9 release dates are more or less timed that the RC and
> >> final will be available for Alpha 3 and Beta 1 respectively).
> >> 
> >> Personal opinions aside, I object to large-scale changes in
> >> release planning after everyone already agreed to the current
> >> Quantal release schedule.
> > 
> > The Kubuntu team are welcome to determine whatever milestones they
> > want - no one is suggesting anything needs to change for the
> > flavors in their testing cadence. I am purely stating how Nick will
> > be working: the only output of this work is assured mandatory test
> > case testing and this testing uncovering any bugs. As I said
> > earlier, this work is independent of whether we remove milestones
> > or not.
> 
> The Kubuntu team wants to have standard alpha and beta images (i.e.
> the system that is in place and currently works for them).
> 
> As a member of the release team, and someone who has seen the current
> system of freeze/milestone release catch image breaking bugs, I want
> the official desktop/server images to be cross-checked by the
> community by the same process that everyone else uses.
> 
> >>> I don't think it is unreasonable for Canonical to focus its
> >>> resources on Ubuntu as opposed to the flavors.
> >> 
> >> To what extent?
> > 
> > To the extent that Canonical provides investment in Ubuntu, as part
> > of this investment is paying Nick's salary, and as Nick's manager I
> > choose how we resource his time and efforts. How we resource Nick
> > is not a community decision.
> 
> I have no argument that Canonical directs Nick's work in whatever way
> we see fit. However, as someone paid by Canonical to ensure the high
> quality of our ARM images, I have a vested interest in knowing if I'm
> covered these efforts.
> 
> >> As it stands, what is proposed will not only hinder but likely
> >> cost considerable QA coverage of the many images that are
> >> "community-supported". It is clear that non-Canonical backed
> >> images simply can not support a rolling-QA testing plan, and
> >> depend on the milestones system.
> >> 
> >> By changing how a minority of how images are being tested, it
> >> will only serve to create confusion, and complications. As a
> >> release manager, am I now to track down each team, make sure from
> >> them that they've met whatever QA schedule they've set for
> >> themselves, and then release off that?
> >> 
> >> As it stands, the vast majority of image testing comes at
> >> milestones, and often picks up people who are otherwise
> >> uninvolved in a flavors development. There has often been calls
> >> to ask for additional testers for X image in #ubuntu-testing
> >> should coverage be lacking, which have been always answered.
> > 
> > As I have said a few times now...If a flavor can't maintain the
> > same level of testing, the flavor can just do the testing it can
> > cope with. This might mean just doing the testing with the current
> > cadence of milestones: just because I am ramping up our manual
> > testing efforts with Nick does not mean anyone else has to change
> > their own testing cadence.
> 
> As you stated later, ARM is not directly apart of these efforts as of
> this moment. As such, the ARM images must remain on the current
> freeze/release process.
> 
> > Flavors are in charge of their own destiny, and this includes
> > testing cadence.
> 
> I believe the practical upshot of this is given that community-flavors
> do not wish to abandon the system that is simply easer to remove
> Ubuntu ISOs off the tracker (or otherwise marked as non-essential/low
> priority) at milestone time and allow everyone else to carry on as is.
> 
> Any community image that wishes to follow in Canonical's stead is
> welcome to bring that up with the release manager, and after making
> sure a proper process of QA reporting is ensured will then be excused
> from the standard ISO tracking (and will not take part of any image
> milestones except final).
> 
> To make sure that we don't release a sub-par product, all flavors
> should undergo full testing at the end of the cycle, with the
> understanding that should they not meet release standards, they can
> and will be dropped.
> 
> >>> To be clear, the testing cadence is up to whatever the flavor
> >>> wants it to be; I am not suggesting we impose this two-week
> >>> cadence on anyone other than me imposing it on Nick. :-)
> >> 
> >> Then why change the existing system?
> >> 
> >> Nothing about the existing system will prevent the Canonical QA
> >> efforts from implementing this.
> > 
> > You are conflating two different things: testing cadence and
> > milestones. To be clear: the cadence of Nick's testing has nothing
> > to do with milestones and whether we choose to have them or not
> > have them. My only point here is that I am disconnecting Nick's
> > cadence from milestones so that if we do choose to not have them in
> > the future, nothing changes.
> 
> No, I'm not. The former is being used to justify the removal of the
> later, and the current system of alpha/betas.
> 
> >>> Naturally our flavors generally don't pay people to coordinate
> >>> this kind of testing (maybe Blue Systems could invest here for
> >>> Kubuntu?), but there is no requirement for the flavors to test
> >>> every two weeks. If you folks want to test once a month or once
> >>> every six weeks, then go ahead and do that. Importantly though,
> >>> as with all flavors, you will need to coordinate this work
> >>> yourselves within your community.
> >> 
> >> How does ARM factor into this?
> >> 
> >> I have already cited the failure by Canonical QA to test the
> >> armhf+omap4 image for precise. Were it not for manual
> >> intervention, and the normal Ubuntu QA testing by both myself and
> >> other members of the community, I'm certain that we would have
> >> shipped an image that was not only substandard, but completely
> >> broken.
> >> 
> >> It should be cited for completeness that this was caught at *Beta
> >> 2*, and only because there were voiced concerns on image quality
> >> that I sat and tested it extensively.
> >> 
> >> Where were the QA efforts by Canonical QA for all the previous
> >> milestones? As it stands, I'm only aware of one person that was
> >> responsible for all ARM images that were supported during
> >> precise.
> >> 
> >> Furthermore, speaking as the ARM Server Tech Lead in Canonical,
> >> the PES team has to  manually QA our own images due to the fact
> >> that in general, we are the only teams with access to the
> >> hardware. We do NOT have the bandwidth required for this
> >> additional testing initiative.
> > 
> > From my teams perspective, we were never asked to test ARM (and
> > thus left it our of Nick's targeting), although this is something
> > that we are going to be helping with moving forward.
> 
> Good to know.
> 
> >>> I see the milestones as quite disconnected from the testing:
> >>> the milestones are a useful means of consolidating efforts
> >>> around *something*, such as targeting work items and deadlines;
> >>> it is just that we happen to use these milestones as a means to
> >>> release. I personally don't see the point of the alpha
> >>> releases...our users don't use them, and our developers are
> >>> running the daily development branch anyway, so to me it makes
> >>> sense to get rid of the milestones at some point.
> >> 
> >> I use milestones extremely regularly. Both to gauge our goals by
> >> comparing what on the image is implemented to what is left and a
> >> common place to gauge any bug fixes between milestones to make
> >> sure we've fixed realistic issues between them. In addition, it
> >> provides a time where a considerable part of the developer
> >> community comes together to provide sanity checks on images.
> > 
> > As has been said before, no-one is suggesting milestones for
> > targeting work are removed....
> 
> The confusion on the term milestone was already cleared up. Just for
> the record, when I say milestone, I consider it milestone images
> (mostly because that's when we confirm during QA testing that
> everything slated for that milestone went in, and wasn't accidently
> mismarked as completed or such).
> 
> >> In addition, as a rule (and excluding precise), during
> >> development cycles, I generally only dist-upgrade my laptop
> >> during freezes from milestone to milestone until after Debian
> >> Import Freeze due to archive breakage and other issues.
> > 
> > ...and here lies the point: you should be able to upgrade
> > *anytime*; not just at certain points. This is the gist behind
> > Rick's points about assuring stability in our development release.
> 
> True, this reason for the milestones existing right now will be
> resolved. That being said, it is only one part of the larger picture,
> and many that do run the development releases upgrade on a much more
> frequent timeframe.
> 
> >> Precise was considerably less broken during development due to
> >> the simple fact we were syncing from Debian testing vs. unstable,
> >> which ensured a minimum of package breaking due to Debian's
> >> strict policies on how packages enter the testing archive.
> > 
> > That is one of the reasons why Precise was less broken, but there
> > was also automated testing, acceptance criteria, increased manual
> > testing, and a stronger focus on QA.
> 
> I'm awaiting an explanation on how a Canonical supported image then
> was nearly released in a broken state at Beta 2.
> 
> One of the biggest hassles of running a development release is
> packages going into an uninstallable state or otherwise skewing.
> Syncing from testing solved this issue since our upstream for the
> period was always in a consistent state and thus we did not have
> issues such as transitions to worry about.
> 
> Once our equivalent of these tools are in place, then yes, we will
> have a development release that can be upgraded both safely and at will.
> 
> >>> My goal in this cycle is to ensure we have a regular testing
> >>> cadence for Ubuntu and not based on milestones; if the Kubuntu
> >>> team want to have your own internal milestones for targeting
> >>> work and testing, I see no reason why you can't do that on your
> >>> own schedule.
> >> 
> >> I can see a very good one. We are forsaking consistency across
> >> all flavors for Canonical's interests. This even affects the
> >> Ubuntu Desktop and Server images on "community-supported"
> >> architectures as there is no one who will be doing this work for
> >> armel or powerpc.
> > 
> > This is not about "Canonicals' interests" this is about building
> > efficiency in how we build Ubuntu; this efficiency benefits not
> > just Canonical staff, but our wider community of participants who
> > can assure a more stable development release.
> > 
> > Nothing is being taken away here: flavors are still free to
> > coordinate their testing as they see fit.
> > 
> > Jono
> 
> Scott has made it clear that if the current system of alpha/betas is
> abolished, they will not be able to manage to maintain their current
> level of quality. Kubuntu (and by extension, the other flavors) depend
> on the current system as is.
> 
> As ARM and PowerPC is not actively part of these new QA efforts, it is
> clear that we are also dependent on the current system to keep quality.
> 
> It should be noted though that as long as one architecture in an image
> can't proscribe to the new testing method, the packageset relating to
> that image in the release pocket must be frozen as we have no way of
> freezing on a per-architecture basis.

+1.

If Ubuntu wants to skip the Alpha/Beta milestones because Canonical thinks 
it's other QA efforts are sufficient, I think they should feel free to do it, but 
the existing structure needs to stay in place for the rest of us.

Scott K