Herd 3 ISO testing post-mortem

[Henrik Nilsen Omma]

Hi,

With Herd 2 we started a forums-based testing team to get more people 
involved in ISO testing. A fair number of tests were reported in forum 
threads. However, it was not clear how we would smoothly combine these 
results with the testing performed by the devel team, traditionally 
registered in the wiki. The wiki page itself has intrinsic scalability 
problems as only one person can edit at a time, even when split over 
several pages.

So at the sprint we agreed to try tracking the ISO testing in Malone 
with a separate Launchpad product, ubuntu-iso-tests. This seems to have 
worked well enough for the fairly casual event of releasing Herd 3 but 
we must decide whether this system can cut it for Beta, RC and Final, 
and what adjustments we need to make.

I collected some feedback as we went along and made a few tweaks. The 
setup used is described here: 
https://wiki.ubuntu.com/Testing/ReportingResults (in howto form)

Feedback:

* It should be easy to see what tests have been performed already, 
ideally on a single page as was possible with the wiki. This can 
probably be done reasonably well using the status field on each test 
report and filtering with tags (so those wishing to track Kubuntu 
results can filter on that tag)

* A dedicated mailing list used for announcements and ISO testing bug 
mail -- Those subscribed would get messages such as 'Start testing now' 
and 'The amd64 images have been rebuilt. Test again.' without having to 
check the web page. The list traffic could also be greped for things 
like bug numbers.

* Users inevitably confuse these tracker bugs with real distro bugs and 
file real bugs for ubuntu in the ubuntu-iso-tests product. That's not a 
major concern as long s the number is small and they can be reassigned 
to distro tasks.


We should look at what short-term tweaks we can make to make this 
approach usable for the Feisty release (more feedback welcome!) and how 
we should manage this in the longer term. I've written some thoughts on 
the latter here: https://wiki.ubuntu.com/Testing/ISOTrackerSpec

Henrik