libc borked (and I stop testing)

[Scott Kitterman]

On Thursday 13 March 2008 08:03:32 Vincenzo Ciancia wrote:
> Il giorno mer, 12/03/2008 alle 23.20 -0400, Cory K. ha scritto:
> > Thanx to the genius who let the libc update through and rendered 3
> > systems unbootable here. I look forward to your visit to my home to fix
> > them.
> >
> > Frustrated and pissed,
> >
> > Cory K.
>
> Even though the tone of the mail is angry, it's really bad that things
> like this libc update happen - I personally don't understand how this is
> possible at all, if developers test their packages.

I've uploaded stuff that turned out to be broken.  There are many possible 
reasons for this.  It's not feasible to do full regression testing for every 
upload.  Not just due to time, but because of the wide range of hardware used 
for Ubuntu.
 
> A "revert system after upgrade" mode should be designed and implemented
> to the benefit of testers (unionfs plus a "commit" operation to the main
> filesystem seems to me like an implementable solution). Would not be
> efficient but would be a bit safer - you already have unionfs in the
> livecd so you have some expertise.

This is the sort of thing that should be proposed as a specification and 
decided on at UDS.  It sounds like a nice idea.  I'm not qualified to have an 
opinion on how feasible it is.

> I am sad to say that my hardy testing experience stops here - I wanted
> to make my experience as a free software user, and as a developer,
> available to ubuntu community as a form of payment for such a good
> distribution. Problem is not the libc bug by itself of course. If you
> want to know read below - but it's not necessary at all.
>
> Problem is that I should waste hours fixing the libc bug, and I am doing
> this just to let the world benefit from fixes I can already install and
> hack up locally on my pc. The balance between costs and benefits is
> dropping down too quick.
>
> Many regressions I've personally been trying to help sorting out have a
> fix, signaled by one of the testers (usually not me since I am not that
> smart, but I usually took the time to test the fix and reported) and the
> fix is not being applied, and developers are waiting for *users* to
> UVFE. I am more and more being convinced that testing new ubuntu is a
> complete waste of time for me.

I've found just the opposite.  I've found as I got more and more involved in 
first testing and then development I've been able to get more and more of my 
personal pain points dealt with.  

Also, keep in mind that most developers are volunteers.  Volunteer work gets 
done on the basis of interest.  If a user wants a problem solved, I've got 
neither the time nor interest in being their personal UVFe (now FFe) writer.  
I'm glad to help them figure out how to do it, but I'm fully busy working on 
the problems that I'm trying to solve.

> The main point that, to my eyes, the ubuntu "upload-enabled community"
> seems not to be understanding, is that one should try to re-use people's
> expertise. You can't ask a person that already can debug a kernel module
> to also learn to package debs and all the ubuntu burocracy. That's a
> problem of developers. If you have a clever user (I am *not* talking
> about me :) ) that provides a fix and explains how he/she got there, you
> can't ask for more. You are the developer, you have the expertise to fix
> bugs in ubuntu, the tester provided the fix, having the expertise to
> test it, why not joining forces?

This is certainly ideal, but it's not like the developer is sitting around 
waiting for more to work on.  What we need are more people working on all 
levels of the problem.  It is a general case that we could use more people 
who know packaging working on packaging up available fixes.  I think there 
have been some recent initiatives to encourage this.

> <personal story follows, the main point of the e-mail is what you
> already read>
>
> Next LTS won't have proper support for my tablet. I surrender. It's two
> years I have been waiting the day I can advice ubuntu to people who have
> the same laptop as mine, and still nobody cares. Next year I will have
> to return this laptop to university, and I'll perhaps buy a different
> tablet. With different problems. And I've never seen ubuntu working out
> of the box there - even though there always was a well-known and
> signaled to developers way to make it work.
>
> I've seen things stopping working, nobody cared in the world. For
> example, my sd card reader worked in edgy and will never work in any
> future ubuntu release. I opened a bug *during feisty beta* - it used to
> work in some feisty alpha but don't know which one, then it was marked
> as duplicate of another bug, which after months was fixed and was not a
> dupe of mine, I then had to reopen a new bug, and *nobody cared
> anymore*. Don't bull*hit on me. The problem I am pointing out is real.
> There is a regression from previous releases and nobody cares, because
> few users have it. But that's a regression. Ok, few users have it
> because the vaste majority of tablet users don't even consider the crazy
> idea of running that hacky linux on it. Accept this and if and when
> ubuntu will work on tablets really out of the box, you'll see how many
> users are affected by such regressions.
>
> I've followed bug reports, provided requested information, tried to
> debug problems, in some rare case even provided the fix personally.
>
> In the past, I uploaded a couple of fixed debs, asked for UVFEs when
> necessary, and even obtained those.
>
> It takes lots of time, especially for one who does it seldomly. You may
> say that people should move *before freeze*. I personally  try to wait
> for developers to ack fixes and set them up for release. This often does
> *not* happen before freeze, so users typically end up not only having to
> learn how to prepare the new changelog, how to properly choose new
> version, how to prepare a debdiff and sign the changes, but also, since
> freeze time is passed, they have to learn UVFEs or other freeze
> exceptions.
> </personal story>

Getting a spec accepted and working with the developers to get it implemented 
is a great way to deal with these kinds of problems.  If you can write up a 
specification (or blueprint) (see 
https://blueprints.launchpad.net/ubuntu/+addspec ), define the requirements, 
and get them agreed at the next UDS (you don't have to be physically present, 
but it helps), then when it comes to the next freeze (if stuff isn't done) 
the rationale is that you want to do X to support implementation of an 
approved spec.  

It also helps with getting people organized around what you are trying to 
accomplish.  None of us know all about everything and tend to specialize.  If 
you have a spec, you can point to it in the bug and that helps motivate 
developers to spend time on it since they see the big picture.

> So I stop testing, get out of my status of half man/half developer and
> get back being an user. If next LTS is broken for my laptop, I'll simply
> fix it locally, as I always did from the age of debian potato.
>
> And if you don't care at all loosing me as a tester, I can understand
> you :) Just delete this e-mail and live happier.

I do care.  It's clear you've got a lot to offer.  It's unfortunate that it 
hasn't always been taken advantage of better.

> A possible idea to improve the situation is to have a "regression" tag,
> and to mark "high priority" all regressions. Say what you want, but this
> is *exactly* the behaviour that one would expect from any software
> distributor: things works, you break it, I tell you as soon as I
> discover it, you fix it as soon as possible because the bug is in the
> change you just made, so your change has to be fixed. If you let the
> regression there for three years, you'll have hysterical raisins when
> you put your hands back on that code.

This sounds reasonable to me.

Scott K