reflecting on first UDS session on "rolling releases"

[Scott Kitterman]

Robert Collins <robertc at robertcollins.net> wrote:

>On 6 March 2013 17:13, Scott Kitterman <ubuntu at kitterman.com> wrote:
>> On Tuesday, March 05, 2013 08:07:36 PM Jono Bacon wrote:
>>> On Tue, Mar 5, 2013 at 7:43 PM, Scott Kitterman
><ubuntu at kitterman.com>
>> wrote:
>>> > What percentage of code in the default install is covered by
>automated
>>> > tests?
>>> I am not sure, maybe the QA team can weigh in on this.
>>>
>>> > For a relatively small project, such approaches are conceivable. 
>For
>>> > something the size of an installed Ubuntu (pick your favorite
>flavor)
>>> > system, I think we're a long ways away from being sufficiently
>>> > instrumented with tests.
>>> I agree: I think automated testing is going to be essential here to
>>> *assure* quality across the system. I think we would need ensure
>that
>>> we have good test coverage across the core components.
>>
>> I agree.  My main point is that such things are pre-requisites to a
>new
>> release model.  We don't have them, so we should get them before we
>change to
>> something we're not ready to support.
>
>I don't see them as strict pre-requisites. Its like this: if you have
>500% coverage you can still ship broken code. (100% coverage is not
>complete coverage, because all (open source) coverage tools todate can
>at most report branch coverage, not domain and range coverage (though
>a http://en.wikipedia.org/wiki/Symbolic_execution based coverage
>analysis would be pretty close). Once you have complete coverage all
>you know is that you have claimed that your code should do what it
>does, *not* that that is correct: you can still violate the
>expectations of third party libraries and that leads to bugs.
>
>What not enough tests *actually* means is that individual developers
>making changes need to think harder. While there is a safety barrier
>of 'noone really uses this' there is no driver for contributors to pay
>attention to detail : they can fix any issue by another upload.
>Exactly what conclusion to draw from that I'm not sure; will leave it
>as an exercise for the reader.

As long as we're integrating externally developed code, we're at the mercy of upstream practices.  While Canonical has done a lot of work around upstream code it develops, that in no way translates across the archive. 

If we can't fully guarantee functionality then code has to be landed somewhere and tested before it should be exposed to end users.

We have made a lot of progress on development release stability, but a few months of successwhile Debian is in freeze aren't enough to know where we are on the quality continuum yet.

Scott K