Let's Discuss Interim Releases (and a Rolling Release)

[David Henningsson]

On 02/28/2013 10:06 PM, Robbie Williamson wrote:
> On 02/28/2013 02:49 PM, David Henningsson wrote:
>> On 02/28/2013 05:09 PM, Martin Pitt wrote:
>>>>    * Keep doing daily quality and keep improving our daily quality.
>>>
>>> Big +1. I'm particularly looking forward to integrating our automatic
>>> package tests with britney.
>>
>> The QA work done in -proposed has increased the productivity for the
>> rest of us, no doubt about that.
>>
>> But still, a word of caution here. Every piece of code even remotely
>> related to the hardware, not only the Linux kernel but also most of the
>> plumbing layer, is quite difficult (or even impossible) to automate
>> testing for. Even if we would set up robots in our lab looking at the
>> screen for artifacts, talking into the microphone and so on, we wouldn't
>> cover the world's hardware.
>>
>> Hardware becomes increasingly complex, diverse, and so testing it takes
>> a lot of time. You can't go test thousands of machines to see if their
>> headphone outputs stopped working every single day.
>>
>> Do we have a plan to deal with those types of bugs?
>
> Maybe I'm missing something, but don't we have this problem *now*,
> regardless of a rolling release or not?  The only way to reasonably
> solve this is get hardware OEMs to participate, who can't even tolerate
> our 6 month cadence and thus do so on the LTS...which isn't changing.

I think you do have a point there, but let me answer from a different 
perspective.

When I was new to Ubuntu, the intuitive thing to do to help out was to 
download a beta release, test it, and report bugs. That's what betas are 
for, right? Well, I learned that if I did that, the developers were 
triaging my bug report around final freeze, and after that there was no 
possibility to change anything. I then tried reporting bugs much 
earlier, all I would get was a report back two months later, telling me 
to test a new version of the package. After a few cycles, I had learned 
that the right time to do testing was around feature freeze; when it's 
still easy to upload, but the upstream versions have stopped pouring in.

As we now move to a rolling release schedule, when is the right time to 
do a wide-scale testing and report bugs? Without just being met with a 
"please check if it's fixed in the next version" message?

There is a difference between "daily quality" and "non-LTS release 
quality", and a wide-scale hardware testing is one of those things that 
make up the difference. This wide-scale hardware testing is not done by 
hardware OEMs, but by the community, at least for the larger part.

And; that wide-scale testing in turn benefits the kernel/X packages we 
backport into the LTS point releases.

That is not to say I'm against moving to a rolling release; from a poll 
on a Swedish news site [1] most of our users seem to like it (73% for, 
27% against), I'm just saying that this is a tricky problem we need to 
solve somehow, if we can.


-- 
David Henningsson, Canonical Ltd.
https://launchpad.net/~diwic

[1] 
http://techworld.idg.se/2.2524/1.488083/darfor-kan-ubuntu-1404-bli-den-sista-pa-lang-tid