Outdated Laptop Test plan makes testing a less pleasant experience

[Daniel Kulesz]

Hello Sergio,

thanks a lot for your comprehensive answer. I will try to further clarify / 
refine some of the mentioned points:

> exc-001: There is no example folder / symlink anymore, the content was
> moved
> 
> > to /usr/share/example-content already some time ago. It is not clear
> > which files / applications are targetted, and what the expected output
> > is. (not only
> > the applications should start correctly, but they should also handle the
> > files
> > correctly)
> 
> In fact I don't know how to handle this, because I don't know actual and
> future plans about this folder.
> Suggestions and comments are welcome.
As far as I understood it, the main goal of the LaptopTesting effort is to do 
a system test for *future* Ubuntu Releases, not for past ones. Therefore - if 
there are any system changes which require changes in existing test cases 
(like with the examples folder) - the changes should be reflected in the test 
plan, so that the testers can execute the test cases on the future (current 
alpha/beta/rc...) pre-releases. Perceiving traceability could be achieved 
simply done by mentioning in the Test Report, against which Test plan 
(=currently that would be the version of the wiki pages) the tests have been 
carried out.

> 
> > snr-001: Entering the password is not needed when testing via LiveCD,
> > this should be mentioned.
> 
> Done.
Looks good. Maybe it should be considered tomark the optional test steps more 
eye-catching, i.e.:
3b. (only applies in Live Environment) You should be presented with a password 
prompt; enter your password

> > snr-002: Executing more than 30 cycles of sleep/wakeup could lead to
> > hardware
> > damage, limiting the cycles to 5 should be really enough!
> 
> This kind of test is present even in checkbox, I don't have an answer about
> this.
> Maybe a member of kernel team could clarify this.
I would appreciate this very much. Could you try to get an answer from the 
kernel team? (I dunno if they read this list)

> > him-001: It is not clear, if this applies only to internal card readers
> > only.
> 
> It should be clear because it's in the "system" section. External card
> reader tests are in "USB" section (hum-*).
Alright, but then some description (just one sentence) of what is meant by 
"system level" tests would be nice in the test plan, maybe with the remark, 
that tests for hardware which is not present shall be skipped.

> 
> > khk-001: The expected behaviour is, that Ubuntu also shows the volume
> > meter on
> > screen; but on many older machines the volume is adjusted, but the user
> > is not
> > given appropiate (visual) feedback. This should be included in this test
> > case
> > as well, to spot the affected machines.
> 
> This test is about checking if multimedia hotkeys do the appropriate action
> after pressing them. (e.g. lowering volume)
> Some laptop hotkeys are not mapped yet and this need to be reported (for
> example).
> The issue you are reporting is a "visual" one and even if it doesn't seem
> to be an hardware problem it should be reported as bug as well, specifying
> it on the bug description.
Well yes, it's a usability problem at first sight. But actually there is 
hardware where the driver supports this visual feedback (i.e. on my Thinkpad 
X301) and there is hardware, where the driver does not support that. I guess 
this is related to ACPI event handling etc. - but since the software already 
supports the visual feedback, it's a system bug if the driver does not notify 
the software about the action that happened in here.

> > hsi-001: the media file "fables_01_01_aesop.spx" seems to be outdated,
> > there
> > is other media included in 10.10; It should be also verified, that the
> > sound
> > plays correctly and does not produce any stottering
> > hhi-001: see hsi-001
> 
> Now it should be ok.
Great!

> > his-001: Changing refresh rate not covered
> > his-001: (video in general) there is no test case to test for fluent
> > video playback (maybe except exc-001)
> 
> We are testing laptops, that should be important using crt monitors.
> About fluent video playback,exc-001 test could be sufficient to cover it.
> In fact, an application test (mediaplayer) should be considered, not an
> hardware one.
This is also a video driver, and therefore a system issue. On some drivers 
(i.e. Intel GM855) the playback only works in normal window but not in 
fullscreen. There you get either a disorted picture, or not the full available 
screen size is used for feedback. So, there are laptops where it works fine 
(with the same media player version and media file) and there are laptops 
where it fails.

> > bfu-001: Not clear, what is meant by app - is this the whole operating
> > system?
> 
> It should be ok now.
Perfect!

> 
> > hpu-002: Ubuntu has various bugs regarding wrong paper size (especially
> > A4 vs.
> > letter) in some drivers, this should be verified here as well.
> 
> As mentioned before, this could be considered in an application test, not
> an hardware one.
> Anyway this shouldn't prevent you or someone else to report and link bugs
> you find doing a test, even if not directly related to it.
> 
Again, this is a printer driver issue and therefore a system level problem. 
With the same version of the applications and the media to be printed it 
prints fine on some printers, while it fails on others. 

> > hpu-002, hds-001, hds-002, hds-003, hds-004 : missing here - at least
> > some of
> > them should be included into LaptopTesting as well, this is pretty
> > important
> > for many mobile users
> 
> Good point.
Lmay

> 
> > Other, general aspects:
> > * the template for the Laptop entries
> > (https://wiki.ubuntu.com/Testing/Laptop/Reports/Template) is *very*
> > incomplete. Most people seem to include info like lshw output, dmidecode
> > details etc. - why is this all missing in the template? Therefore,
> > instead of
> > re-using the template, testers have to copy/paste the structure around
> > from existing reports.
> 
> This is because the original template included just a table, then some
> people start adding other info on their own without mentioning in ML or
> changing template.
> For Natty I'm going to enrich it with other data to be filled in by users.
I suggest to browse the existing reports and to merge back "good looking" 
sectioning into the template. It seems that most users are relying on one of 
the past reports, i.e. "lazychris2000" used some of the elements I used in my 
Thinkpad T30 report, like i.e. the global machine specs - which is very good, 
but it's the job of the template to provide this info, not the job of the 
tester to visit several reports and merge in structural elements.

> > * the main wiki page (https://wiki.ubuntu.com/Testing/Laptop/Reports) has
> > no
> > consistency regarding the naming of the tested "editions"
> 
> What do you mean?
> Putting the edition name in that page should be sufficient.
Well I don't really see a naming convention from those existing titles:
- Ubuntu 10.10 Maverick Meerkat 
- Ubuntu 10.10 Live 
- Ubuntu 10.10 (Maverick Meerkat) Desktop amd64 
- Ubuntu 10.10 Desktop i386 

There should be some defined format, how to name things, i.e.
<Ubuntu|Kubuntu|Xubuntu...> <Version> <arch><Live|Desktop> 

Reports which do not follow this convention could be reformatted to match it, 
i.e. by replacing unknown Information with question marks. Example:

Ubuntu 10.10 Live
=>
Ubuntu 10.10 <?arch> Live

> > It would make testing Ubuntu (Pre-)Releases a much more pleasant
> > experience,
> > if you could update the test plan and look into the mentioned issues. I
> > strongly believe there are much more people out there who would be
> > willing to
> > contribute to systematic Laptop Testing, but finding the wiki pages and
> > test
> > plan in the current (not very useable) state could be pretty frustrating.
> 
> Thank you very much Daniel for your suggestions.
> The laptop testing is a community project so everyone can help and
> contribute updating testcases and suggesting improvements etc.
> In the future feel free to keep on sharing your thoughts and proposing your
> ideas here in ML so we can discuss together.
After having almost gave up hope that there is someone who maintains these 
wiki pages (apart from the "pure" testers), I am very happy to have received 
your answer and the reactions in the test plans!

> P.S.: Apart from issues you mentioned, I can say that the main problem of
> this project is the use of wiki pages to report results.
> I proposed a blueprint to integrate the laptop testing in the ISO tracker,
> and it'll be discussed at the Ubuntu Developer Summit for Natty [¹]
Very good idea. If I understood it correctly, it looks more or less like a 
webified version of the system testing tool "Justus" (written in Java) we 
developed at our institute (http://justus.tigris.org/ there is also a newer 
version available at http://sourceforge.net/projects/justus2/). I fear that 
implementing such a tool as a web application requires *a lot* of effort, but 
of course I would welcome the integration of such a feature in launchpad!

Another possible solution to this problem I have thought about was doing it 
the other way round: Integrating an export/import functionality for wikitext 
in Justus, so testers could download/upload testplans directly from/to the 
Wiki. The big pro of the Wiki is that it's easy to read, update and browse for 
"non-testers" and "potential new testers", while experienced testers could 
benefit from the comfort of a dedicated desktop application.

Again, thank you very much for your answer Sergio. Now I am motivated again to 
do some more Laptop Tests for Natty, once the first Alpha becomes available.

Regards, Daniel