Mandatory / Optional testcases

Brian Murray brian at ubuntu.com
Mon Oct 11 22:04:31 UTC 2010


On Thu, Sep 30, 2010 at 01:12:58PM +0200, Ara Pulido wrote:
> 
> Hello all!
> 
> This email is about the way we manage testcases during milestone release
> testing cycles (also known as ISO testing). If you don't have experience
> with ISO testing you can read the documentation [1].
> 
> *A bit of background*
> 
> From Beta milestone, onwards, the release team requires the testcases in
> the ISO tracker to be covered in a 100% ratio, in order to approve the
> release. During the Lucid cycle we (Release team & QA) decided to move
> some of those testcases as "Optional" for release, as they were not that
> important, and we needed better coverage in the rest.
> 
> The problem was that most of the optional testcases remained uncovered.
> During Maverick UDS, we had a session to rethink the testcases and move
> some of them back to mandatory, and the other way round. Also, the
> optional testcases were not that optional anymore, they had to be run at
> least once every milestone (as opposed to mandatory, that need to be run
> at least once in every image that contains the testcase in every milestone).
> 
> During the Maverick cycle we have had an obvious problem with the
> naming. People not aware of that session still thought that the optional
> testcases were optional. Some others, when they discovered that the
> optional testcases were not that optional, missed having optional testcases.
> 
> I think that those optional (really optional) tests have a value. Maybe
> they are not release critical but they are good to have around, so that
> people can manage better their testing processes. Some examples: the ARM
> team likes to have a Netboot testcase, but it is not critical if not
> covered; the Ubuntu Studio team would like to have some HW specific
> tests, but still make it to release, if they are not covered; etc.
> 
> *Proposal*
> 
>  * Create a new category "Run Once" and move there those testcases that
> need to be run at least once every milestone. (Most of the tests that
> are now under the "optional" category.)
>  * Use the "Optional" category for those tests that really are optional
> for release. If they are not covered, or they fail, the release can
> still go.
>  * The "Mandatory" category would remain as it is.
> 
> This is a proposal for the Natty release. If the release team agrees
> with those changes, I could implement them during the first week of the
> new cycle. If you think that this should be discussed during UDS, let me
> know.

This makes a lot of sense to me and I don't any further discussion is
necessary.

-- 
Brian Murray
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-quality/attachments/20101011/f716781c/attachment.pgp>


More information about the Ubuntu-qa mailing list