From brian at ubuntu.com Thu Sep 1 22:24:52 2022 From: brian at ubuntu.com (Brian Murray) Date: Thu, 1 Sep 2022 15:24:52 -0700 Subject: Updating Lubuntu's testcases In-Reply-To: References: Message-ID: <20220901222452.GH2300191@murraytwins.com> On Mon, Aug 29, 2022 at 11:17:55AM +1000, Chris Guiver wrote: > Lubuntu's QA is mostly handled on > https://phab.lubuntu.me/w/release-team/testing-checklist/ with a note > at the top that doc is supposed to be unnecessary after > https://phab.lubuntu.me/T56 which pushes us (Lubuntu) back to > iso.qa.ubuntu.com... > > fyi: most install tests are done by Leó (Leokolb) & myself; and > personally I like our checklist as I can choose the oldest test > performed & redo/update it.. instead of a fresh page for each daily... > > We have some 'updated' checklists, created long ago which never > completed review. > > 1. Is there a guide to the formatting, ie. I wanted more than one > line at the top so created two lines... (emphasis?) but is there > somewhere where what I-can-do | I-cannot-do is defined? I gather > labels are DL = descriptive.list, DT = description term within DL, DD= > description & EM = emphasis/strong As far as I know there is not a guide to the formatting and looking at the admin portion of the iso.qa.ubuntu.com site the test cases sections says "some html is allowed" which isn't terribly helpful. However, I tried multiple lines and that did work. I am also happy to try other experiments as necessary. > 2. Tests end up MANDATORY or OPTIONAL, where is that set? That is set in the admin portion of the site when creating a "testsuite" of test cases. > 3. Is there a tool where I can view the created testcase in somewhat > PREVIEW state (without codes) so I can re-read & hopefully detect > errors? Not at this point in time but I tried copying and pasting a test case into the body section at https://www.w3schools.com/html/tryit.asp?filename=tryhtml_intro and it worked okay. However, the numbered list and bullet points did not appear. So maybe that isn't useful. > 4. Is there a guide for reviewers I can read? > > Walter (wxl) originally created the list in our QA checklist; the > issue is we now have lots (a guide to understanding them I created > here - https://discourse.lubuntu.me/t/testing-checklist-understanding-the-testcases/2743), > and I sure don't want them all mandatory. I also consider the FOUR > BIOS installs as roughly equivalent (variations of encryption, > internet & swap), our FOUR EFI installs the ~same, as well as FOUR > Secure-uEFI.. To counter this I've a testcase [script] that attempts > to get a tester to select one & run it (so one bios can be mandatory, > one uefi mandatory, one secure-uefi..) but that's also more complex > than the four testcases each script replaced.. It's not clear to me exactly what you are asking for here. However, I'd much rather there be separate test cases for each different scenario than have people choose an installation type and enter which one they chose in a comment. The latter would make it harder for people to know which ones are left to test and for the release team to know which tests are outstanding. I imagine you were trying to reduce duplicating the same test cases with minor variations (install type) but I think my next comments address that. > Any advice or direction would be appreciated. I do want to mention some changes that Dave made to the Raspberry Pi test cases that are quite useful though. In the definitions folder[1] of ubuntu-manual-tests there is a pi_desktop_cases.xml file which contains a series of tests and then has multiple test case ids which reference those tests. A script[2] is then run to generate the test case files which are put into the iso.qa.ubuntu.com site. This reduces the amount of duplicate information and the need to update multiple test case files e.g. if something changes in the installer you can update it in the xml file and generate new test cases instead of having to edit each test case. If you are going to be working on adding new test cases I'd strongly suggest starting with this new format. [1] https://git.launchpad.net/ubuntu-manual-tests/tree/definitions [2] https://git.launchpad.net/ubuntu-manual-tests/tree/tools/test_case_gen Cheers, -- Brian Murray