Releasing Alphas and Betas without "freezing"

[Michael Casadevall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/20/2012 11:14 PM, Jono Bacon wrote:
> On Wed, Jun 20, 2012 at 9:24 PM, Scott Kitterman
<ubuntu at kitterman.com> wrote:
>> On Wednesday, June 20, 2012 09:13:17 PM Jono Bacon wrote:
>>> On Wed, Jun 20, 2012 at 8:52 PM, Scott Kitterman
<ubuntu at kitterman.com>
>> wrote:
>>>>> For our current Ubuntu ISOs. Flavors currently are
>>>>> coordinating
their
>>>>> own testing efforts. They could either latch into the two
>>>>> week cadence, or use their own cadence if desired.
>>>> 

All flavors as it stands right now are actually coordinated to a
common testing and QA cycle based around milestones, as set forth by
the release manager and the manifest. At each milestone, all images,
regardless of backing are tested. If they fail they're removed from
the manifest, and are excluded from the release.

>>>> I find it somewhat unfortunate that the "community" testing
>>>> efforts exclude the community sponsored flavors in the Ubuntu
>>>> project.  I
would
>>>> have hoped that the community team was not just about
>>>> Canonical's products.
>>> 
>>> This shouldn't be a particularly big surprise; Canonical
>>> supports our flavors with infrastructure, but we primarily
>>> focus our engineering and community team staff members on
>>> Ubuntu.

No one is objecting to additional QA efforts dedicated to Canonical
images. That being said, I've yet to see a stated reason on why this
additional QA drive requires changing the milestone process. As Scott
clearly points out what is being proposed will change the
QA/milestone/release process will cause a fair bit of large grief for
all flavors.

I have regularly tested those images and others such as Studio and
Edubuntu on x86 and ports architectures when I have additional cycles
to spend, and spent several cycles making sure Kubuntu's armel images
were in a good state until hardware became more prevalent.

>>> 
>>> If we had more resources we would love to provide help for the 
>>> flavors, and we are certainly happy to offer any guidance and
>>> advice, with with our current resources and staffing, Nick
>>> doesn't have the bandwidth to handle more the than the Ubuntu
>>> ISOs and associated testing. Saying that, I know Nick is in
>>> contact with many of the flavors to ensure they get the support
>>> they need to set up their own comprehensive testing plans.
>> 

The first I heard of any of this was when this email was sent to
u-devel. To be frank, changes of this magnitude should have been
discussed in seasons at UDS-Q, and not in hallway conversations where
many of the affected parties were not present.

Many images such of Kubuntu have worked to have the various milestones
and deadlines set in ways that allow them to incorperate the correct
versions of their upstream packages (i.e. KDE 4.9 release dates are
more or less timed that the RC and final will be available for Alpha 3
and Beta 1 respectively).

Personal opinions aside, I object to large-scale changes in release
planning after everyone already agreed to the current Quantal release
schedule.

>> Perhaps I misunderstood, but I thought that you were saying this
was about the
>> community team he had organized to support ISO testing.  Nothing
>> to
do with
>> Canonical resources.  I think that such a team should not be
focused on just
>> Canonical products.
>> 
>> As a Kubuntu developer trying to get Kubuntu images tested for
milestones,
>> I've often gotten a lot of help from Canonical people in Ubuntu
>> QA.  I appreciate that.  That's not what I'm talking about.
> 
> Well to be clear, Nick is one member of the Ubuntu community who
> is focused on testing. Other people are welcome to coordinate
> testing campaigns and get others interested and excited about
> testing, but Nick's focus is explicitly on the Ubuntu ISOs. Of
> course, if community members want to volunteer to help test the
> flavors then that is great.
> 
> I don't think it is unreasonable for Canonical to focus its
> resources on Ubuntu as opposed to the flavors.
> 

To what extent?

As it stands, what is proposed will not only hinder but likely cost
considerable QA coverage of the many images that are
"community-supported". It is clear that non-Canonical backed images
simply can not support a rolling-QA testing plan, and depend on the
milestones system.

By changing how a minority of how images are being tested, it will
only serve to create confusion, and complications. As a release
manager, am I now to track down each team, make sure from them that
they've met whatever QA schedule they've set for themselves, and then
release off that?

As it stands, the vast majority of image testing comes at milestones,
and often picks up people who are otherwise uninvolved in a flavors
development. There has often been calls to ask for additional testers
for X image in #ubuntu-testing should coverage be lacking, which have
been always answered.

>> None of the non-Canonical flavors have the resources to pick up
>> the
pace of ISO
>> testing.  If Canonical is insistent on reorganizing the
>> development
process in
>> a way that works better for them (dropping milestones and more
>> frequent testing), you're going to leave us behind.
>> 
>> Fundamentally the development and testing model has to be
>> something
that the
>> entire project can support and I don't think it's unreasonable
>> to
expect that
>> the community team person assigned to work on QA would be trying
>> to
build a
>> team of community members to accomplish that.
> 
> To be clear, the testing cadence is up to whatever the flavor wants
> it to be; I am not suggesting we impose this two-week cadence on
> anyone other than me imposing it on Nick. :-)
> 

Then why change the existing system?

Nothing about the existing system will prevent the Canonical QA
efforts from implementing this.

> My mail earlier in this thread was merely making it clear that from
> my teams perspective, we are assigning resources on a two-week
> cadence. Now, admittedly, a big reason why we can do this is
> because we pay Nick to help coordinate this work.
> 
> Naturally our flavors generally don't pay people to coordinate
> this kind of testing (maybe Blue Systems could invest here for
> Kubuntu?), but there is no requirement for the flavors to test
> every two weeks. If you folks want to test once a month or once
> every six weeks, then go ahead and do that. Importantly though, as
> with all flavors, you will need to coordinate this work yourselves
> within your community.
> 

How does ARM factor into this?

I have already cited the failure by Canonical QA to test the
armhf+omap4 image for precise. Were it not for manual intervention,
and the normal Ubuntu QA testing by both myself and other members of
the community, I'm certain that we would have shipped an image that
was not only substandard, but completely broken.

It should be cited for completeness that this was caught at *Beta 2*,
and only because there were voiced concerns on image quality that I
sat and tested it extensively.

Where were the QA efforts by Canonical QA for all the previous
milestones? As it stands, I'm only aware of one person that was
responsible for all ARM images that were supported during precise.

Furthermore, speaking as the ARM Server Tech Lead in Canonical, the
PES team has to  manually QA our own images due to the fact that in
general, we are the only teams with access to the hardware. We do NOT
have the bandwidth required for this additional testing initiative.

> I see the milestones as quite disconnected from the testing: the 
> milestones are a useful means of consolidating efforts around 
> *something*, such as targeting work items and deadlines; it is
> just that we happen to use these milestones as a means to release.
> I personally don't see the point of the alpha releases...our users
> don't use them, and our developers are running the daily
> development branch anyway, so to me it makes sense to get rid of
> the milestones at some point.
> 

I use milestones extremely regularly. Both to gauge our goals by
comparing what on the image is implemented to what is left and a
common place to gauge any bug fixes between milestones to make sure
we've fixed realistic issues between them. In addition, it provides a
time where a considerable part of the developer community comes
together to provide sanity checks on images.

In addition, as a rule (and excluding precise), during development
cycles, I generally only dist-upgrade my laptop during freezes from
milestone to milestone until after Debian Import Freeze due to archive
breakage and other issues.

Precise was considerably less broken during development due to the
simple fact we were syncing from Debian testing vs. unstable, which
ensured a minimum of package breaking due to Debian's strict policies
on how packages enter the testing archive.

> My goal in this cycle is to ensure we have a regular testing
> cadence for Ubuntu and not based on milestones; if the Kubuntu team
> want to have your own internal milestones for targeting work and
> testing, I see no reason why you can't do that on your own
> schedule.
> 

I can see a very good one. We are forsaking consistency across all
flavors for Canonical's interests. This even affects the Ubuntu
Desktop and Server images on "community-supported" architectures as
there is no one who will be doing this work for armel or powerpc.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJP40WuAAoJEHM+GkLSJHY5lg0P/3XszACvOqnle4L+Qx95wD0U
hGS8uS2dc72PzmYAAKEkNyU5YnD706AY7dyU440zC/MrBLcrxky2QiPL8LNyuowR
ZII+1KH5TAffDVru/NpJbOAZVAmghXmJJYVRpiWuLZfpFDcnAKXf4pyrfG5TzIz6
dxsejTMMWMsXJz07PmfHxXMgZXpQnT6d0K+k6eNB2QoxsbPKjRp8tdJUkwdElYsL
t2Vz7w/7cAEwl8Tf4LK5Vz4ti/NcIztLKRW6R3HBCdp14iK/Pon3Xe3eNycyfWZH
K2EHosqJEna8Ftktb+gbPABV54tH+MwcKUv3RbWLS3EK6TYI8xNGbJg5d02MQp9j
x+3K2zGPDced3NDFScI/OHZF0QmS3OfWp8fU0wPldjvYopYcn8IRc+YzPeYU0uKv
PPfAm0FkuxQ3aV8St5V9BLZO/+llKkFECSOfW4SKnjQ3zUG6VxjTmQPDlisrEM4W
U5oJqXAiFyGFbBI+KSiGB9zPmjH2sR9qNhZJlMZbfNY2unRn1I2kW/y8dJiz110e
GbS+UAN9OTtE5IsQBIYbhIq4gMWY0jx7b72TXUvcRJmfsmsLEveHgJIq4f/j5X1H
CnzaZUGAgUgGqiivx7kghB7Fb33ktPDXYO6NryLjFRLTtSsphzn73ePhYCw1cSn4
tz+PBu2bgheTDRuYB8Ri
=oXTd
-----END PGP SIGNATURE-----