Better targeted ISO testing

[Henrik Nilsen Omma]

Hi all,

 From recent RC ISO testing I think we should adjust testing as follows: 
Do ISO testing in phases, first focusing on uncovering bugs then on 
validation of images.

Currently we start preparing a major milestone by freezing the archive 
and only letting in changes related to fixing bugs targeted for that 
milestone (in theory). Even those uploads can contain bugs of course, 
but when the number of uploads is limited it is easier to track what 
changes caused them. We then wait until we think we have a reasonable 
set of milestone candidate images and proceed with the full battery of 
tests.

One problem with that is that it is difficult to motivate testers to 
test lots of early images when, from experience, we know that they are 
likely to not really to be milestone candidates, because we will find 
some bugs as we go. It becomes: "Please test these images today, knowing 
that they will be redone 2-3 times, and come back tomorrow and test the 
same thing. kthxbye."

At the same time, we have a real need to have the images tested early, 
because there are very likely to be a range of bugs in there. I believe 
the solution is to test images in two (or more) stages.

In the first stage we should aim for the widest possible code coverage 
with fewer tests and we should do this right after the milestone freeze. 
The purpose of this is to uncover release critical bugs, not to validate 
the CDs for release. This requires a sensible mix of archs, CD types and 
install methods, but not full coverage of everything. Once those bugs 
are discovered we can get to work on fixing them without being too tight 
up against the milestone release.

By the time we start the second stage we should be more confident that 
all the show-stoppers have been found and we can focus on real 
validation testing on all CDs, as well as noting non-release critical 
bugs for the future. This will involve a larger group of people and 
would run 1-2 days.

While it's late to change methods now, I think we should follow this 
approach even for gutsy final. That means starting bug-search testing 
right after RC release and the broader validation testing likely on 
Monday the 15th. Our test tracker isn't designed to handle this kind of 
phased-in testing ATM, but we can make it work with a wiki page.

Henrik