reflecting on first UDS session on "rolling releases"
robertc at robertcollins.net
Wed Mar 6 04:34:19 UTC 2013
On 6 March 2013 17:13, Scott Kitterman <ubuntu at kitterman.com> wrote:
> On Tuesday, March 05, 2013 08:07:36 PM Jono Bacon wrote:
>> On Tue, Mar 5, 2013 at 7:43 PM, Scott Kitterman <ubuntu at kitterman.com>
>> > What percentage of code in the default install is covered by automated
>> > tests?
>> I am not sure, maybe the QA team can weigh in on this.
>> > For a relatively small project, such approaches are conceivable. For
>> > something the size of an installed Ubuntu (pick your favorite flavor)
>> > system, I think we're a long ways away from being sufficiently
>> > instrumented with tests.
>> I agree: I think automated testing is going to be essential here to
>> *assure* quality across the system. I think we would need ensure that
>> we have good test coverage across the core components.
> I agree. My main point is that such things are pre-requisites to a new
> release model. We don't have them, so we should get them before we change to
> something we're not ready to support.
I don't see them as strict pre-requisites. Its like this: if you have
500% coverage you can still ship broken code. (100% coverage is not
complete coverage, because all (open source) coverage tools todate can
at most report branch coverage, not domain and range coverage (though
a http://en.wikipedia.org/wiki/Symbolic_execution based coverage
analysis would be pretty close). Once you have complete coverage all
you know is that you have claimed that your code should do what it
does, *not* that that is correct: you can still violate the
expectations of third party libraries and that leads to bugs.
What not enough tests *actually* means is that individual developers
making changes need to think harder. While there is a safety barrier
of 'noone really uses this' there is no driver for contributors to pay
attention to detail : they can fix any issue by another upload.
Exactly what conclusion to draw from that I'm not sure; will leave it
as an exercise for the reader.
Robert Collins <rbtcollins at hp.com>
HP Cloud Services
More information about the ubuntu-devel