Exit Criteria to Production and Release Blocker Nominations

[Null Ack]

Gday folks,

In my previous email I discussed my concepts for the importance of
static analysis on code in any robust test effort and my view of
needing to introduce mandatory release entry criteria from upstream
that includes a minimum standard benchmark with static analysis
results. Like many of us, as much as I spend a substantial amount of
my personal time executing runtime testing on Ubuntu builds, there is
clear practical limits to manual test coverage and there is more
efficient approaches to some types of bugs.

In this email I'd like to discuss the other end of our release
methodology, the exit criteria to production.

Recently I was involved in a bug that presented an error dialogue
within the gnome system log to the user each time the app was run. I
nominated this issue as a release blocker for Jaunty. I was shocked to
see this nomination declined, and when queried as to why, I was
advised that "the nominations are there to mark blockers for jaunty,
that issue now is
a minor one happening for early jaunty users only and has no reason to
be set as a blocker."

Please appreciate this isn't a shot at the developer in question. I
consider him to be a friend and have a high regard for his work. When
I look at the situation objectively I think we need to better develop
policy on what release blockers are and how they can be rightly
declined.

It's true that in this specific case the issue was fixed before
production, but from a process perspective another incident might see
it slip through. I agree with the developer that the issue of having
an error dialogue window be displayed to the user by default each time
an app is run is technically low in a functional sense, but the
perception it gives about quality is problematic indeed.

When Jordan was the leader of the QA team, he asked me my thoughts on
how to measure quality. My response was one that I've been considering
throughout my career. As the years have rolled on I have come to
understand that quality can't be empirically measured, it can only be
treated as the perception of quality that is unique to the observer
who has unique sets of wants, needs, expectations, world view, prior
experiences and so forth.

What I've learnt is that it takes allot of work to bring software to
the point where most people have the perception its a quality thing,
and it's very easy to loose that perception of quality.

One key way of loosing the perception of quality is having software
that wiggs out and presents errors to the user, however functionally
trivial, each time it is run by default.

Right now, it seems interpretations of justification for declining
blockers means its ok to goto production with default error messages
being presented to the user.

We need to more carefully consider the policy in light of the way it
effects the user experience and the perception of quality. I suspect
many release nominations are being declined because of time and
resource constraints rather than other criteria, which is plain wrong
and counter to our strategic objectives.

Regards

Nullack