Monthly Updates versus Monthly Images

[Scott Kitterman]

On Tuesday, March 05, 2013 06:34:29 AM Adam Conrad wrote:
> TL;DR summary: Monthly updates are harmful, monthly images are cool, let's
> do the latter without turning them into the former and all frolick happily
> in fields of time, money, and cheesecake.
> 
> So, I'm intentionally breaking the thread here to shift gears slightly. I
> have no intention of discussing the merits of rolling versus not, of going
> on about whether now is the right time to switch, or if we need a year of
> discussion before we can consider it, or any such thing.  This is purely
> on the topic of so-called "monthly updates" versus monthly images, and my
> position that the latter could prove quite useful, while the former could
> be actively harmful.

If one were going to do rolling, I mostly agree with this.

> We seem to be approaching this by conflating the two concepts of releases
> and images as if they're both the same thing and I think we can all take a
> step back and agree that, in fact, they're not.  If we choose to have a
> monthly milestone planning cadence for work items and release a monthly
> image to wrap marketing and messaging around, there's definitely value in
> that.  Being able to announce to the world that we had these seven major
> new features land in Xubuntu this month, these two awesome new things in
> Kubuntu, these three in Lubuntu, etc, is quite nice, and probably much less
> chaotic than having everyone plan to different cadences and sporadically
> tell the world about things as they happen.  Keeping us all on a similar
> track has always proven quite helpful for interaction between teams and
> flavours, and I see significant value in that continuing.
> 
> The above stated, I see no reason why this planning cadence and image cycle
> needs to, in any way, look like a "release", and I think it's actively
> harmful, both to our development velocity (hey look, a buzzword, you won't
> hold it against me, right?) and to our users.
> 
> For starters, let's look at all the things I think this monthly image can't
> be, if we want to maintain a lightweight process that doesn't slow us all
> down and make small children weep:
> 
>  1) We can't be doing mini freeze-and-release cycles before each image is
>     released.  I've seen it come up several times in the thread, and this
>     suggestion will just slow us to a crawl.  It's actively undoing any
>     benefit we'd get from moving to a rolling release cycle.

We've done two day migration freezes for the relevant bits of the archive to 
support Alpha 1 and Alpha 2 in this cycle.  I think short migration freezes 
like this (in contrast to the traditional upload freeze of 5 - 7 days) has had 
almost no impact on development velocity.  I've found it helpful in making 
sure we're getting a good set of images for a milestone.

I think a one or two day migration freeze would be a useful thing that 
wouldn't harm the goals associated with rolling in any meaningful way.

>  2) We can't be providing out-of-band critical bugfix support for these
>     images.  How they ship is how they are, and we need to trust that the
>     preliminary smoketesting on them before they're released is enough to
>     ensure an install/reboot cycle and that we won't be destroying people's
>     hardware, but that's about as far as we can go.
> 
> In short, we can't be wasting engineering time on the images themselves,
> or they become "releases", and everything that comes with that.  The more
> time we spend on the images, the less time we spend on the rolling release
> itself, and the lower the quality of the LTS that comes out of the other
> end of this meat grinder.
> 
> So, looking at various solutions proposed for this dilemma, we see people
> suggesting clever use of -updates and -security to attempt to simulate a
> monthly release, we see people suggesting phased updates dialed down to
> zero, and many other curious options.  Each of these makes the assumption
> that monthly updates are a good thing.  Each of these ignores the following
> facts, as far as I can tell:
> 
>  1) If a nasty bug is discovered in the rolling release a day after we put
>     out a monthly, we either need to revisit our "no SRUs to the monthlies"
>     policy on a case-by-case basis, or we ask users to live with that bug
>     for ~29 days until the tools say "hey, cool, you can upgrade now".  In
>     the current default setup, that bug would have been fixed in seven days
>     or less, as that's the default update-manager nag frequency.
> 
>  2) Security support either needs a stable base to build on (the "freeze
>     the release pocket for a month" model), or it can't be provided out-of-
>     band.  Having the security pocket depend on things that we might need
>     to remove from the release pocket is problematic.
> 
> These two points feel like they're at odds to me, as freezing the world to
> allow an out-of-band security model means making users live with every non-
> security bug for a month until the next set of bugs are dropped on their
> heads.  And, conversely, providing out-of-band security support without the
> freeze model (which has been suggested) is a bit of a nightmare from the
> archive management standpoint with, as far as I can tell, very little gain.
> 
> All of these things in mind, I'd like to propose the following set of ideas,
> and see if we can massage them into something palatable to everyone's
> goals:
> 
>  1) No freezes, no pre-release process, monthly images are merely a fully
>     smoketested daily being released on a predetermined date, no better or
>     worse than the previous or next smoketested daily, just what happens
>     to be in the archive on that date.  We target our work items to that
>     date, and if they slip, so be it, the features that make it in can then
>     be mentioned in messaging for 13.05, 13.06, etc, those that don't can
>     be postponed for the next month.

I'd suggest a migration freeze from when the first image is built until testing 
is done.  These do need to actually support doing installs, not eat hardware, 
etc. and so I think holding migrations to make it possible to address critical 
issues (and  mean critical), respin, and move on will be useful and minimally 
invasive.

>  2) No out-of-band support at all, SRU or security.  The only slight change
>     from how we do things now would be that security updates destined for
>     the development release would be built in the security PPA (which does
>     not build against -proposed), so they don't pick up new dependencies
>     and can then be copied to the archive and not accidentally get caught
>     up in library transition snags that hinder their migration to the
>     release pocket.

If this is going to be messaged as anything other than the development series, 
I think that rolling needs to be included in USNs.  I don't think it's 
appropriate to leave rolling out and leave users with the impression it isn't 
getting security fixes when we've told them it's ~safe to use.

>  3) We twiddle the phased-updates spec a teensy bit so that P-U-P values
>     over 100 are treated by update-manager as security/critical updates,
>     and offered immediately, rather than after the configured update delay,
>     much as packages in the -security pockets are now offered.  With this
>     model, we can make the scripts that copy from the security PPA to the
>     archive set the phased update probability to 101 for security uploads,
>     and have them treated as "special" by update manager, without having to
>     actually use the -security pocket and deal with the annoyance of a
>     pocket that doesn't have a stable base to depend on.
> 
>  4) We leave the update-manager default at seven days (which I feel is a
>     perfectly sane compromise between "getting bugfixes out quickly" and
>     "not bugging users constantly or eating their bandwidth"), but add an
>     option for "every four weeks", for people who really feel that the
>     current top-end value of two weeks is still too frequent for them.
> 
> Again, I can't stress enough that I feel that artificially delaying updates
> to users to create the illusion that we're "releasing" once a month is not
> doing anyone any favours, not us or them.  We'll be getting bug reports on
> "ancient" versions, and constantly asking people to muck with whatever
> config is necessary to get off the monthly track and on to the rolling one
> just to get their fix, we'll be asking users to run software that none of
> us will be running, because we all want/need to have up-to-date systems to
> develop against, and we'd have no one but ourselves to blame for this state
> of affairs, as we're the ones who created the belief that updates only
> happen once a month, and that bugs are just something you live with until
> the next magical code drop four weeks away.
> 
> If we were going to freeze every month to stabilize, and offer proper SRU
> bugfixes to those mini releases, I'd be all for monthly updates, except for
> the part where our community, both paid and unpaid, would need to be about
> twice as large to accomodate such lofty goals.  As it stands, I see no way
> that we can offer a monthly release without shafting both ourselves and our
> users.
> 
> This may well read as a disjointed and potentially grumpy view of the world,
> and I imagine such an interpretation wouldn't be far off, but I feel quite
> strongly that the idea of withholding updates from users isn't just a poor
> decision but is, in fact, actively harmful.  I know I have colleagues who
> agree with me on that point, and I've seen a few speak up in the previous
> threads.
> 
> With that in mind, I'd like to ask that conversations around monthly updates
> shift from being "how do we provide monthly updates" to "how do we meet our
> needs without providing monthly updates".  The two greatest needs I see are
> security support and giving users the option for lower frequency of less
> critical updates.  I think leveraging phased-updates to that end is a
> fairly simple, lightweight, and elegant solution that completely avoids the
> harm and complexity of any of the proposed monthly updates schemes.

I think for phased updates to be meaningful in this context it would have to 
be a bit different.  As (IIRC) Highvoltage mentioned, he wants to be at the 
back of the iceberg:

http://hellenicpolytheist.files.wordpress.com/2012/07/killer_whale_eating_penguins-
jump-in-air.jpg

What's needed is not only show me updates every X days, but only show me non-
critical updates that are at least X days old.  I don't think we have a way to 
express that.

Scott K