Releasing Alphas and Betas without "freezing"

[Jono Bacon]

On Thu, Jun 21, 2012 at 9:02 AM, Michael Casadevall
<mcasadevall at ubuntu.com> wrote:
>>>> If we had more resources we would love to provide help for the
>>>> flavors, and we are certainly happy to offer any guidance and
>>>> advice, with with our current resources and staffing, Nick
>>>> doesn't have the bandwidth to handle more the than the Ubuntu
>>>> ISOs and associated testing. Saying that, I know Nick is in
>>>> contact with many of the flavors to ensure they get the support
>>>> they need to set up their own comprehensive testing plans.
>>>
>
> The first I heard of any of this was when this email was sent to
> u-devel. To be frank, changes of this magnitude should have been
> discussed in seasons at UDS-Q, and not in hallway conversations where
> many of the affected parties were not present.

What changes of magnitude? We had already made it pretty clear that my
team is not focused on flavors (historically we have never focused on
flavors in our committed work items and always on Ubuntu), and all I
am outlining here is the cadence in how Nick we be conducting his
work. This is purely a way in which I choose to manage my resources to
serve Ubuntu best.

Yes, these plans were not discussed at UDS, because they have evolved
since UDS, but Nick quite clearly laid out his testing strategy at UDS
(which is not to dissimilar from this...he had scheduled testing plans
for milestones and interim dailies).

> Many images such of Kubuntu have worked to have the various milestones
> and deadlines set in ways that allow them to incorperate the correct
> versions of their upstream packages (i.e. KDE 4.9 release dates are
> more or less timed that the RC and final will be available for Alpha 3
> and Beta 1 respectively).
>
> Personal opinions aside, I object to large-scale changes in release
> planning after everyone already agreed to the current Quantal release
> schedule.

The Kubuntu team are welcome to determine whatever milestones they
want - no one is suggesting anything needs to change for the flavors
in their testing cadence. I am purely stating how Nick will be
working: the only output of this work is assured mandatory test case
testing and this testing uncovering any bugs. As I said earlier, this
work is independent of whether we remove milestones or not.

>> I don't think it is unreasonable for Canonical to focus its
>> resources on Ubuntu as opposed to the flavors.
>>
>
> To what extent?

To the extent that Canonical provides investment in Ubuntu, as part of
this investment is paying Nick's salary, and as Nick's manager I
choose how we resource his time and efforts. How we resource Nick is
not a community decision.

> As it stands, what is proposed will not only hinder but likely cost
> considerable QA coverage of the many images that are
> "community-supported". It is clear that non-Canonical backed images
> simply can not support a rolling-QA testing plan, and depend on the
> milestones system.
>
> By changing how a minority of how images are being tested, it will
> only serve to create confusion, and complications. As a release
> manager, am I now to track down each team, make sure from them that
> they've met whatever QA schedule they've set for themselves, and then
> release off that?
>
> As it stands, the vast majority of image testing comes at milestones,
> and often picks up people who are otherwise uninvolved in a flavors
> development. There has often been calls to ask for additional testers
> for X image in #ubuntu-testing should coverage be lacking, which have
> been always answered.

As I have said a few times now...If a flavor can't maintain the same
level of testing, the flavor can just do the testing it can cope with.
This might mean just doing the testing with the current cadence of
milestones: just because I am ramping up our manual testing efforts
with Nick does not mean anyone else has to change their own testing
cadence.

Flavors are in charge of their own destiny, and this includes testing cadence.

>> To be clear, the testing cadence is up to whatever the flavor wants
>> it to be; I am not suggesting we impose this two-week cadence on
>> anyone other than me imposing it on Nick. :-)
>>
>
> Then why change the existing system?
>
> Nothing about the existing system will prevent the Canonical QA
> efforts from implementing this.

You are conflating two different things: testing cadence and
milestones. To be clear: the cadence of Nick's testing has nothing to
do with milestones and whether we choose to have them or not have
them. My only point here is that I am disconnecting Nick's cadence
from milestones so that if we do choose to not have them in the
future, nothing changes.

>> Naturally our flavors generally don't pay people to coordinate
>> this kind of testing (maybe Blue Systems could invest here for
>> Kubuntu?), but there is no requirement for the flavors to test
>> every two weeks. If you folks want to test once a month or once
>> every six weeks, then go ahead and do that. Importantly though, as
>> with all flavors, you will need to coordinate this work yourselves
>> within your community.
>>
>
> How does ARM factor into this?
>
> I have already cited the failure by Canonical QA to test the
> armhf+omap4 image for precise. Were it not for manual intervention,
> and the normal Ubuntu QA testing by both myself and other members of
> the community, I'm certain that we would have shipped an image that
> was not only substandard, but completely broken.
>
> It should be cited for completeness that this was caught at *Beta 2*,
> and only because there were voiced concerns on image quality that I
> sat and tested it extensively.
>
> Where were the QA efforts by Canonical QA for all the previous
> milestones? As it stands, I'm only aware of one person that was
> responsible for all ARM images that were supported during precise.
>
> Furthermore, speaking as the ARM Server Tech Lead in Canonical, the
> PES team has to  manually QA our own images due to the fact that in
> general, we are the only teams with access to the hardware. We do NOT
> have the bandwidth required for this additional testing initiative.

>From my teams perspective, we were never asked to test ARM (and thus
left it our of Nick's targeting), although this is something that we
are going to be helping with moving forward.

>> I see the milestones as quite disconnected from the testing: the
>> milestones are a useful means of consolidating efforts around
>> *something*, such as targeting work items and deadlines; it is
>> just that we happen to use these milestones as a means to release.
>> I personally don't see the point of the alpha releases...our users
>> don't use them, and our developers are running the daily
>> development branch anyway, so to me it makes sense to get rid of
>> the milestones at some point.
>>
>
> I use milestones extremely regularly. Both to gauge our goals by
> comparing what on the image is implemented to what is left and a
> common place to gauge any bug fixes between milestones to make sure
> we've fixed realistic issues between them. In addition, it provides a
> time where a considerable part of the developer community comes
> together to provide sanity checks on images.

As has been said before, no-one is suggesting milestones for targeting
work are removed....

> In addition, as a rule (and excluding precise), during development
> cycles, I generally only dist-upgrade my laptop during freezes from
> milestone to milestone until after Debian Import Freeze due to archive
> breakage and other issues.

...and here lies the point: you should be able to upgrade *anytime*;
not just at certain points. This is the gist behind Rick's points
about assuring stability in our development release.

> Precise was considerably less broken during development due to the
> simple fact we were syncing from Debian testing vs. unstable, which
> ensured a minimum of package breaking due to Debian's strict policies
> on how packages enter the testing archive.

That is one of the reasons why Precise was less broken, but there was
also automated testing, acceptance criteria, increased manual testing,
and a stronger focus on QA.

>> My goal in this cycle is to ensure we have a regular testing
>> cadence for Ubuntu and not based on milestones; if the Kubuntu team
>> want to have your own internal milestones for targeting work and
>> testing, I see no reason why you can't do that on your own
>> schedule.
>>
>
> I can see a very good one. We are forsaking consistency across all
> flavors for Canonical's interests. This even affects the Ubuntu
> Desktop and Server images on "community-supported" architectures as
> there is no one who will be doing this work for armel or powerpc.

This is not about "Canonicals' interests" this is about building
efficiency in how we build Ubuntu; this efficiency benefits not just
Canonical staff, but our wider community of participants who can
assure a more stable development release.

Nothing is being taken away here: flavors are still free to coordinate
their testing as they see fit.

   Jono

-- 
Jono Bacon
Ubuntu Community Manager
www.ubuntu.com / www.jonobacon.org
www.identi.ca/jonobacon www.twitter.com/jonobacon