Backtracing, Invalidated Bugs and Quality

[Null Ack]

Evening Devs,

Tonight I was doing some of my test suite and I had the
tracker-preferences crash unexpectedly doing routine workflow with
viewing (not changing) preferences. Apport came through and I ended up
at an invalid existing bug from 2007 because the user had not
submitted debugging symbols. This has happened to me before and my
mind has been busy since with thinking about how this detracts from
quality and what to do about it. These are real bugs, some of them are
in production, that are not being fixed.

I'm not convinced that the strategy of asking users to install
specialised debugging packages is the right way to go. I see a very
low hit rate with this working in practice. I have professional
experience in managing testing projects and consulting on related
fields so with Ubuntu being close to my heart I often think about how
we approach testing and what might be processes that could be
improved. Can I please offer some thoughts:

1. The Debug By Default Build. This would be where the entire
operating system is built using debug packages. This could be at a
targeted point of the lifecycle, such as during Alpha, where apport
will deliver all debug symbols by default. We could still distribute a
non debug build for users who must have that type of build, but it
could be hidden away so that the most common type is the debug build.
We engage some community evangilists who promote the importance of it
so it gets readily brought into practice.

2. The Hybrid Debug Build. Similar, but for technical reasons only
some packages are debug builds.

3. Extending Investment at the Canonical Test Lab. There is sound and
proven arguments I could help to present that demonstrate the cost to
fix defects as they progress in the lifecycle, both in terms of
monetary costs as well as costs to things like image, future sales and
so forth - how it increases at an escalating rate the further on it
progresses in the lifecycle. A business case could be built that looks
at extending whatever Canonical Test Lab exists now with the mission
for capturing the higher priority backtracing bugs and replicating
them in house under controlled conditions. My consulting career has
been based in Australia and I only have knowledge of what the rates
are for various testing roles in my country. I do though understand
that some other countries have far lower rates. I'm not suggesting
exploitation of cheap labour but I am suggesting that the labour costs
could be reduced considerably with choosing a location for the lab at
fair market prices. It might be additionally possible to setup a large
scale test lab using donated hardware from partners to Canonical. A
more aggressive strategy could be looked at into the future, such as
having planned multiple phases. Like another team looking at building
the automated test harness up to the point where most function points
are tested all night, every night, in every build before it gets
posted to daily ISOs. The results could be datamined by automatic
processes that then engage Ubuntu developers or upstream projects
automatically.

4. Extending The Ubuntu Entry Criteria. At least from my perspective
(which maybe insular) the practice of the Ubuntu release methodology
for eligibility of new code into existing packages is something along
the lines of has Debian accepted it and does it compile. I fully
understand that upstream projects lack person power with runtime
testing and they need their code to be included in pre-release
distro's to be tested. One thing that has gotten results for me in
projects I've managed is not just focusing on runtime level tests.
Static testing tools really can be useful and can be quite specific.
It's possible to set arbitary benchmarks for release entry criteria as
a minimum standard. You can set levels of compliance such as mandatory
where certain code problems are specifically banned, and others with
an allowed number of warnings and so on. I realise this would need
careful implementation but I think chipping away at it piece by piece
could realistically over time became an accepted part of what upstream
projects do in a standard way to demonstrate their new code changes
are ready for distros to look at.

These are just some ideas I had. Anyway, sincerely thanks for Ubuntu
and all the work that goes into it.

Regards

Nullack