Atlanta Linux Fest Hardware testing

[Leann Ogasawara]

On Sun, 2009-09-20 at 17:41 -0500, Manoj Iyer wrote:
[snip]
> 111 machines were tested, and the raw test logs are available at 
> http://people.canonical.com/~manjo/usbkey/atlfest_results/
> Users also filed bugs on failures on the field.

Unfortunately after parsing the results, it looks like 61 of the
machines were unsuccessful in saving any results.  From what I've been
hearing, it sounds like testers may have prematurely removed the usb
sticks prior to the results actually being written by checkbox.  For the
future, it would likely be better to sync results after each test rather
than all at once at the end if possible.

I've subsequently published the remaining Pass/Fail/Skip results [1].
You can drill down for a summary of machines which either
passed/failed/skipped each test.  For the failures, if a tester happened
to report a bug which we were able to link to their hw (ie the reporter
also submitted their hardware information to launchpad), a link to the
bug number is provided.  It unfortunately appears that only 4 bugs were
filed that the reporter submitted their hw profile to launchpad which we
could then positively link to results.  I've tagged these bugs
"atlfest2009" [2].  I have found examples, such as bug 433193 [3], where
we are able to see from the bug description that this was reported
during the ATL Linux Fest testing but the reporter did not submit their
hw profile to launchpad.  I'm trying to find an alternative way to get
these outlying bugs.  It doesn't appear that checkbox is able to log the
bug numbers that were filed during testing.  Checkbox does seem to have
the ability to now add a string to the bug description containing the hw
submission id (even if they don't submit their actual hw profile to
launchpad), however it seems this was not available in the version of
checkbox that was used in the usb stick images.  It also would have been
helpful if these bugs would have been have uniquely tagged "atlfest2009"
at the time the bug was reported.

> The logs needs to be parsed and a database needs to be created linking the 
> failures to specific bits of hardware. I think this will help us identify 
> the drivers that will need work to get Karmic support these hardware bits 
> better.

This should already be possible using the existing HWDB in Launchpad
[4].

Thanks,
Leann

[1] http://qa.ubuntu.com/reports/ogasawara/atlfest/results.html
[2] https://bugs.launchpad.net/ubuntu/+bugs?field.tag=atlfest2009
[3] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/433193
[4] https://edge.launchpad.net/+apidoc/#hwdb