Updating Lubuntu's testcases

Brian Murray brian at ubuntu.com
Thu Sep 1 22:24:52 UTC 2022

On Mon, Aug 29, 2022 at 11:17:55AM +1000, Chris Guiver wrote:
> Lubuntu's QA is mostly handled on
> https://phab.lubuntu.me/w/release-team/testing-checklist/ with a note
> at the top that doc is supposed to be unnecessary after
> https://phab.lubuntu.me/T56 which pushes us (Lubuntu) back to
> iso.qa.ubuntu.com...
> fyi: most install tests are done by LeĆ³ (Leokolb) & myself; and
> personally I like our checklist as I can choose the oldest test
> performed & redo/update it.. instead of a fresh page for each daily...
> We have some 'updated' checklists, created long ago which never
> completed review.
> 1.   Is there a guide to the formatting, ie. I wanted more than one
> line at the top so created two <em> lines... (emphasis?) but is there
> somewhere where what I-can-do | I-cannot-do is defined?  I gather
> labels are DL = descriptive.list, DT = description term within DL, DD=
> description & EM = emphasis/strong

As far as I know there is not a guide to the formatting and looking at
the admin portion of the iso.qa.ubuntu.com site the test cases sections
says "some html is allowed" which isn't terribly helpful. However, I
tried multiple <em> lines and that did work. I am also happy to try
other experiments as necessary.

> 2.   Tests end up MANDATORY or OPTIONAL, where is that set?

That is set in the admin portion of the site when creating a "testsuite"
of test cases.

> 3.   Is there a tool where I can view the created testcase in somewhat
> PREVIEW state (without codes) so I can re-read & hopefully detect
> errors?

Not at this point in time but I tried copying and pasting a test case
into the body section at
https://www.w3schools.com/html/tryit.asp?filename=tryhtml_intro and it
worked okay. However, the numbered list and bullet points did not
appear. So maybe that isn't useful.

> 4.   Is there a guide for reviewers I can read?
> Walter (wxl) originally created the list in our QA checklist; the
> issue is we now have lots (a guide to understanding them I created
> here - https://discourse.lubuntu.me/t/testing-checklist-understanding-the-testcases/2743),
> and I sure don't want them all mandatory. I also consider the FOUR
> BIOS installs as roughly equivalent (variations of encryption,
> internet & swap), our FOUR EFI installs the ~same, as well as FOUR
> Secure-uEFI.. To counter this I've a testcase [script] that attempts
> to get a tester to select one & run it (so one bios can be mandatory,
> one uefi mandatory, one secure-uefi..) but that's also more complex
> than the four testcases each script replaced..

It's not clear to me exactly what you are asking for here. However, I'd
much rather there be separate test cases for each different scenario
than have people choose an installation type and enter which one they
chose in a comment. The latter would make it harder for people to know
which ones are left to test and for the release team to know which tests
are outstanding. I imagine you were trying to reduce duplicating the
same test cases with minor variations (install type) but I think my next
comments address that.

> Any advice or direction would be appreciated.

I do want to mention some changes that Dave made to the Raspberry Pi
test cases that are quite useful though. In the definitions folder[1] of
ubuntu-manual-tests there is a pi_desktop_cases.xml file which contains
a series of tests and then has multiple test case ids which reference
those tests. A script[2] is then run to generate the test case files
which are put into the iso.qa.ubuntu.com site. This reduces the amount
of duplicate information and the need to update multiple test case files
e.g. if something changes in the installer you can update it in the xml
file and generate new test cases instead of having to edit each test
case. If you are going to be working on adding new test cases I'd
strongly suggest starting with this new format.

[1] https://git.launchpad.net/ubuntu-manual-tests/tree/definitions

Brian Murray

More information about the Ubuntu-quality mailing list