No subject

the ISO tracker to be covered in a 100% ratio, in order to approve the
release. During the Lucid cycle we (Release team & QA) decided to move
some of those testcases as "Optional" for release, as they were not that
important, and we needed better coverage in the rest.

The problem was that most of the optional testcases remained uncovered.
During Maverick UDS, we had a session to rethink the testcases and move
some of them back to mandatory, and the other way round. Also, the
optional testcases were not that optional anymore, they had to be run at
least once every milestone (as opposed to mandatory, that need to be run
at least once in every image that contains the testcase in every milestone).

During the Maverick cycle we have had an obvious problem with the
naming. People not aware of that session still thought that the optional
testcases were optional. Some others, when they discovered that the
optional testcases were not that optional, missed having optional testcases.

I think that those optional (really optional) tests have a value. Maybe
they are not release critical but they are good to have around, so that
people can manage better their testing processes. Some examples: the ARM
team likes to have a Netboot testcase, but it is not critical if not
covered; the Ubuntu Studio team would like to have some HW specific
tests, but still make it to release, if they are not covered; etc.

*Proposal*

 * Create a new category "Run Once" and move there those testcases that
need to be run at least once every milestone. (Most of the tests that
are now under the "optional" category.)
 * Use the "Optional" category for those tests that really are optional
for release. If they are not covered, or they fail, the release can
still go.
 * The "Mandatory" category would remain as it is.

This is a proposal for the Natty release. If the release team agrees
with those changes, I could implement them during the first week of the
new cycle. If you think that this should be discussed during UDS, let me
know.

Thanks!
Ara.

[1]  http://qa.ubuntu.com/testing/iso-testing/