Exit Criteria to Production and Release Blocker Nominations

[Mike Rooney]

On Tue, Mar 10, 2009 at 9:38 AM, Null Ack <nullack at gmail.com> wrote:
> Gday folks,
>
> In my previous email I discussed my concepts for the importance of
> static analysis on code in any robust test effort and my view of
> needing to introduce mandatory release entry criteria from upstream
> that includes a minimum standard benchmark with static analysis
> results. Like many of us, as much as I spend a substantial amount of
> my personal time executing runtime testing on Ubuntu builds, there is
> clear practical limits to manual test coverage and there is more
> efficient approaches to some types of bugs.
>
> In this email I'd like to discuss the other end of our release
> methodology, the exit criteria to production.
>
> Recently I was involved in a bug that presented an error dialogue
> within the gnome system log to the user each time the app was run. I
> nominated this issue as a release blocker for Jaunty. I was shocked to
> see this nomination declined, and when queried as to why, I was
> advised that "the nominations are there to mark blockers for jaunty,
> that issue now is
> a minor one happening for early jaunty users only and has no reason to
> be set as a blocker."


As a user and tester of alpha versions of Ubuntu, I personally accept
that if a bug is fixed for new users, and it is non-trivial to fix for
previous alpha users, then the time is better spent fixing bugs that
users of the final release will encounter.

I would agree that a pre-release bug which doesn't exist in a fresh
Jaunty install should not be a release blocker. It may mean I have to
re-install or perform a manual work around, but as a tester I am
willing to accept issues like this (and people who aren't probably
shouldn't be using alphas with such expectations), and think it is
better for my time to be spent here and developers time to spent
fixing bugs visible in the upcoming release, then the alternative.
Naturally if a user comes up with a patch to cleanly work around the
issue, it should be considered, although this can still result in
crufty upgrade scripts and such.

In your perspective, it seems like you want the developers/OS to be a
service to the testers, and not the other way around.


> Right now, it seems interpretations of justification for declining
> blockers means its ok to goto production with default error messages
> being presented to the user.

Only when being presented to early alpha users, which is the case as
above if I understand. If there is a huge issue which affects most
users of the RC which would require a painful workaround/reinstall for
example, then I think the issue potentially warrants addressing.


>
> We need to more carefully consider the policy in light of the way it
> effects the user experience and the perception of quality. I suspect
> many release nominations are being declined because of time and
> resource constraints rather than other criteria, which is plain wrong
> and counter to our strategic objectives.


Yes, the quality of a fresh install of the final release, or upgrade
from a supported version.

Anyway that's just my philosophy of being a tester, but I personally
don't want developers spending time on bugs only alpha testers will
experience and not release users, when there are more important things
on their plate. In other words, I don't want my testing to result in
work that isn't potentially improving the quality of the final
release, otherwise I would feel that I am taking away from the release
and not contributing.

Are we on different pages with anything, or am I misunderstanding your
views in some way?

-- 
Michael Rooney
mrooney at gmail.com