Monthly Updates versus Monthly Images

[Adam Conrad]

TL;DR summary: Monthly updates are harmful, monthly images are cool, let's
do the latter without turning them into the former and all frolick happily
in fields of time, money, and cheesecake.

So, I'm intentionally breaking the thread here to shift gears slightly. I
have no intention of discussing the merits of rolling versus not, of going
on about whether now is the right time to switch, or if we need a year of
discussion before we can consider it, or any such thing.  This is purely
on the topic of so-called "monthly updates" versus monthly images, and my
position that the latter could prove quite useful, while the former could
be actively harmful.

We seem to be approaching this by conflating the two concepts of releases
and images as if they're both the same thing and I think we can all take a
step back and agree that, in fact, they're not.  If we choose to have a
monthly milestone planning cadence for work items and release a monthly
image to wrap marketing and messaging around, there's definitely value in
that.  Being able to announce to the world that we had these seven major
new features land in Xubuntu this month, these two awesome new things in
Kubuntu, these three in Lubuntu, etc, is quite nice, and probably much less
chaotic than having everyone plan to different cadences and sporadically
tell the world about things as they happen.  Keeping us all on a similar
track has always proven quite helpful for interaction between teams and
flavours, and I see significant value in that continuing.

The above stated, I see no reason why this planning cadence and image cycle
needs to, in any way, look like a "release", and I think it's actively
harmful, both to our development velocity (hey look, a buzzword, you won't
hold it against me, right?) and to our users.

For starters, let's look at all the things I think this monthly image can't
be, if we want to maintain a lightweight process that doesn't slow us all
down and make small children weep:

 1) We can't be doing mini freeze-and-release cycles before each image is
    released.  I've seen it come up several times in the thread, and this
    suggestion will just slow us to a crawl.  It's actively undoing any
    benefit we'd get from moving to a rolling release cycle.

 2) We can't be providing out-of-band critical bugfix support for these
    images.  How they ship is how they are, and we need to trust that the
    preliminary smoketesting on them before they're released is enough to
    ensure an install/reboot cycle and that we won't be destroying people's
    hardware, but that's about as far as we can go.

In short, we can't be wasting engineering time on the images themselves,
or they become "releases", and everything that comes with that.  The more
time we spend on the images, the less time we spend on the rolling release
itself, and the lower the quality of the LTS that comes out of the other
end of this meat grinder.

So, looking at various solutions proposed for this dilemma, we see people
suggesting clever use of -updates and -security to attempt to simulate a
monthly release, we see people suggesting phased updates dialed down to
zero, and many other curious options.  Each of these makes the assumption
that monthly updates are a good thing.  Each of these ignores the following
facts, as far as I can tell:

 1) If a nasty bug is discovered in the rolling release a day after we put
    out a monthly, we either need to revisit our "no SRUs to the monthlies"
    policy on a case-by-case basis, or we ask users to live with that bug
    for ~29 days until the tools say "hey, cool, you can upgrade now".  In
    the current default setup, that bug would have been fixed in seven days
    or less, as that's the default update-manager nag frequency.

 2) Security support either needs a stable base to build on (the "freeze
    the release pocket for a month" model), or it can't be provided out-of-
    band.  Having the security pocket depend on things that we might need
    to remove from the release pocket is problematic.

These two points feel like they're at odds to me, as freezing the world to
allow an out-of-band security model means making users live with every non-
security bug for a month until the next set of bugs are dropped on their
heads.  And, conversely, providing out-of-band security support without the
freeze model (which has been suggested) is a bit of a nightmare from the
archive management standpoint with, as far as I can tell, very little gain.

All of these things in mind, I'd like to propose the following set of ideas,
and see if we can massage them into something palatable to everyone's goals:

 1) No freezes, no pre-release process, monthly images are merely a fully
    smoketested daily being released on a predetermined date, no better or
    worse than the previous or next smoketested daily, just what happens
    to be in the archive on that date.  We target our work items to that
    date, and if they slip, so be it, the features that make it in can then
    be mentioned in messaging for 13.05, 13.06, etc, those that don't can
    be postponed for the next month.

 2) No out-of-band support at all, SRU or security.  The only slight change
    from how we do things now would be that security updates destined for
    the development release would be built in the security PPA (which does
    not build against -proposed), so they don't pick up new dependencies
    and can then be copied to the archive and not accidentally get caught
    up in library transition snags that hinder their migration to the
    release pocket.

 3) We twiddle the phased-updates spec a teensy bit so that P-U-P values
    over 100 are treated by update-manager as security/critical updates,
    and offered immediately, rather than after the configured update delay,
    much as packages in the -security pockets are now offered.  With this
    model, we can make the scripts that copy from the security PPA to the
    archive set the phased update probability to 101 for security uploads,
    and have them treated as "special" by update manager, without having to
    actually use the -security pocket and deal with the annoyance of a
    pocket that doesn't have a stable base to depend on.

 4) We leave the update-manager default at seven days (which I feel is a
    perfectly sane compromise between "getting bugfixes out quickly" and
    "not bugging users constantly or eating their bandwidth"), but add an
    option for "every four weeks", for people who really feel that the
    current top-end value of two weeks is still too frequent for them.

Again, I can't stress enough that I feel that artificially delaying updates
to users to create the illusion that we're "releasing" once a month is not
doing anyone any favours, not us or them.  We'll be getting bug reports on
"ancient" versions, and constantly asking people to muck with whatever
config is necessary to get off the monthly track and on to the rolling one
just to get their fix, we'll be asking users to run software that none of
us will be running, because we all want/need to have up-to-date systems to
develop against, and we'd have no one but ourselves to blame for this state
of affairs, as we're the ones who created the belief that updates only
happen once a month, and that bugs are just something you live with until
the next magical code drop four weeks away.

If we were going to freeze every month to stabilize, and offer proper SRU
bugfixes to those mini releases, I'd be all for monthly updates, except for
the part where our community, both paid and unpaid, would need to be about
twice as large to accomodate such lofty goals.  As it stands, I see no way
that we can offer a monthly release without shafting both ourselves and our
users.

This may well read as a disjointed and potentially grumpy view of the world,
and I imagine such an interpretation wouldn't be far off, but I feel quite
strongly that the idea of withholding updates from users isn't just a poor
decision but is, in fact, actively harmful.  I know I have colleagues who
agree with me on that point, and I've seen a few speak up in the previous
threads.

With that in mind, I'd like to ask that conversations around monthly updates
shift from being "how do we provide monthly updates" to "how do we meet our
needs without providing monthly updates".  The two greatest needs I see are
security support and giving users the option for lower frequency of less
critical updates.  I think leveraging phased-updates to that end is a fairly
simple, lightweight, and elegant solution that completely avoids the harm
and complexity of any of the proposed monthly updates schemes.

... Adam