LTS and release methodology

[Matt Zimmerman]

On Mon, Jul 07, 2008 at 10:43:44AM -0500, Luke L wrote:
> http://ubuntuforums.org/showthread.php?t=769112
> 
> NOW people realize that something is wrong with the dev cycle!
>
> Introduction
> Ubuntu is in the top tier of Linux distributions. Even since I started using it (6.10), a great amount of progress has been made on improving its usability and compatibility with hardware. These are important steps in allowing it to become a mainstream workstation and server operating system. That said, the development team at Ubuntu should consider a permanent change in its development process as a way to manage its troubles with stability and error regression.

There is always room for improvement, and we are constantly revisiting our
processes to meet the ever-changing and expanding expectations of Ubuntu.
As Ubuntu becomes more widely used by more types of users, it becomes a
greater challenge to maintain a good level of satisfaction for everyone, but
we want to do the best possible job.

I'm glad to see thoughtful opinions being expressed here, and there are a
few things which I'd like to clarify in response to your post (quoted below):

> Regression
> Ubuntu will not be adopted as a desktop OS by end-users or businesses if they have to be consistently concerned that the product has debilitating errors. Even more so, they do not want to have to worry that a new version of the OS will have errors in programs that were stable in previous releases! See Note(2) for an informative post that makes my point. Attention needs to be paid that things DO NOT get broken that were working perfectly before!

No one likes regressions.  A loss of functionality can be a crippling
setback for a casual user, and in something as complex as a Linux
distribution, they are inevitable.  Even when they are not the result of a
bug (for example, an application changing to work differently than it had
before), it's inconvenient and unnecessarily disruptive.  This is one reason
why we offer multiple maintenance cycles.

Casual users should have no need to upgrade their operating system every six
months.  While we, as free software enthusiasts, enjoy the fast pace and
adapting to change, this is not true of most computer users, and most people
(and organizations) are better served by a slower upgrade cycle which
introduces change less frequently.

This is what LTS is all about.  However, even LTS cannot guarantee that
every feature will be retained.  Much of this is beyond our control, as I'll
explain.

> Stability in software
> Why is it that 8.04 “LTS” has such a wave of new features and new versions of software that have not been time-tested to be stable? LTS releases (meant to be exceptionally stable) should not have so many additions to its feature set.
> For example, Firefox 3 is BETA! One of the most used programs in the distro is in the “stable” release in a relatively untested, thus unstable, state. Yes, Firefox 2 has its problems, but none that are critical or unknown. It has been time tested. “Beta” should be nowhere in a stable release.

Just as "newer" doesn't imply "better", "older" doesn't necessarily
mean "more stable" or "more maintainable".  Firefox is a perfect example of
this dilemma.

We didn't choose Firefox 3 because we wanted the latest features---quite the
opposite!  While Firefox 2 may have been more mature at the time of release,
it's also nearing two years of age and is scheduled for mothballing in
December, just eight months after Ubuntu 8.04.

With Ubuntu Desktop LTS scheduled for three years of maintenance, Firefox 3
was the only reasonable choice, paradoxical though it may seem.  If you've
read the source code for Firefox security patches, you'll understand why!

While the initial version we included may have had some rough edges, we're
in better shape for the long term by staying aligned with Mozilla.

> OpenOffice.org 2.4 has what should be considered a critical error, where its Hebrew version displays Arabic numbers instead of decimal numbers (See Note(3) for link). Imagine the English version showing base 10 numbers half the time, base 8 the other half. Wouldn't that be a deal breaker for many users of OOo, and thus many new users of Ubuntu? That is exactly what this could be to many Hebrew users. Again, in an LTS release, OpenOffice 2.3.1 should have been used. New versions don't always mean better versions.

The corresponding bug report is at
https://bugs.launchpad.net/ubuntu/+source/openoffice.org/+bug/210204

While 8.04 may have initially released with this bug, Launchpad shows that
it was fixed for the 8.04.1 update.  Anyone who was unable to use 8.04 for
this reason merely had to wait a couple of months for 8.04.1.

This sounds like it was a very serious bug for Hebrew users, but that
doesn't mean that 2.3.1 would have been a better overall choice.  It had
many other bugs which were fixed in 2.4, most of which affected users
regardless of their native language.

Every one of these choices is a judgment call, weighing risk and benefit to
decide the best course.  OpenOffice.org 2.4.x was used and tested
extensively in the Ubuntu community for months before this bug was reported
in Launchpad, showing that it worked well enough for many other people.
Perhaps not enough of them used a Hebrew configuration, in which case the
question is not "why did you choose OpenOffice.org 2.4?" but "how can we, as
a community, provide better testing for Hebrew users?" 

> Considerations for an LTS
> One idea to prevent such a rush of higher version numbers and new gadgets from breaking a distro is to use a “STS” release as an LTS. For example, Freeze the software versions in 7.10 for use in 8.04, and work on fixing every error reported in those 6 months before the LTS release. This would allow the “beta” of the LTS to be based on an official release, with the testing and feedback of every user of the OS. This would next apply to the 9.04 release, in preparation for the 9.10 LTS (assuming Ubuntu keeps its 6-18 month dev cycle).
> This solution may seem extreme for a distribution on the “bleeding edge”, but if Ubuntu plans to get respect in the mainstream, a balance must be found between development and stability. Currently, that balance is too far toward the former.

This is a very difficult balance to strike, and we have considered many
approaches (including the one you describe above).  None of them are
perfect, and the tradeoffs are complex.

Ubuntu thrives on the efforts of open source developers, and their attention
is very focused on the latest code.  Many projects won't even accept bug
reports if they do not pertain to the latest release.  Most new releases we
incorporate fix many more problems than they create.

Remember, Ubuntu is completely free to everyone.  The primary reason that is
possible is that we "stand on the shoulders of giants".  The other side of
the coin is that we are highly dependent on the efforts of others, over
which we have little or no control.

We aim to provide users with the best experience we can under these
unorthodox circumstances.  We believe it is possible for that experience to
be as good, and better, than what is possible with proprietary software, but
this is a journey we have only just begun over the past four years.  There
are still many unresolved questions about how open source can be molded into
consumer-grade products and what the limitations are, and we, as a
community, are still learning what it takes.

Note that the next LTS will be 10.04, not 9.10.  See
http://www.ubuntu.com/products/ubuntu/release-cycle for more information on
the Ubuntu release cycle.

> Conclusion
> Ubuntu must stop insisting on being on the bleeding edge of features and software if they want to have a “low-error” operating system. This applies even more so for LTS, and this paper is directed toward my disappointment in the QC of this LTS release. Let us briefly review my points, as they apply to any “stable” release of Ubuntu:
> 
> --Regression is unacceptable. Care should be taken that software updates and new features do not break existing functions that were stable previously.

This is easy to say, but consider carefully what it would mean in practice.
How could we implement such a policy in Ubuntu?  Before we can even begin to
estimate the effort required in order to achieve this, we would need to
rigorously specify every feature in Ubuntu and how it should work.  While
this is common in traditional software development models, consider the
sheer scope of Ubuntu: how long would it take, and how many people, just to
enumerate all of this functionality?

Instead, we focus on defining a subset of functionality which can be tested
in practice.  You can find the corresponding test plans here:
https://wiki.ubuntu.com/Testing along with instructions for how you can
participate in the testing effort and find the problems which matter to you.

> --New software should not be included simply because it is new, quite the opposite: new software should rarely included. Firefox beta and OOo 2.4 are notable examples.

I can't agree with you on this point.  In my experience with open source
software, I have nearly always been better off with current software.
Ubuntu was founded with the idea of delivering this to users.

Newer software may introduce some new problems, and it may fix some.
However, software which remains unchanged will never improve.

> --Serious debate should take place as to how LTS releases are developed (same as normal ones, or as “error-fixed” previous releases).
> 
> I hope those in charge of Ubuntu's development take notice of these concerns. I took the time to write this paper (Yes, I wrote it on paper first) for two reasons: my respect for Ubuntu as possibly the easiest and most functional Linux OS in existence, and my sincere worry that if changes are not made in Ubuntu's development process and quality control, this terrific effort will continue to be riddled with errors, and growth of the userbase will be stifled for years.

This mailing list is the right place to discuss this subject.

> My personal issues:In beta, I was not able to get Atheros wifi drivers working (wrapper or native). In stable: Using an ATI 3870 with binary drivers, I cannot use Compiz and play games at the same time; Compiz (non-stable software) breaks my game window. Currently, I cannot get my mic to work on my Realtek ALC882 device. Wine is giving me a lot of trouble, but I cannot fault Ubuntu for that.

It is worthwhile to consider for which issues you blame Ubuntu, and which
you consider the responsibility of another project.  It is by exploring
these expectations and the reasons for them that we will eventually come to
a common understanding quality in open source.

-- 
 - mdz