Interpreting test results across test runs

Chris Gregan chris.gregan at canonical.com
Thu Nov 4 20:02:05 UTC 2010


On 11/03/2010 12:22 PM, Marc Tardif wrote:
> Hi folks,
>
> I would appreciate your feedback about interpreting test results across
> multiple test runs. For example, consider the following test results for
> the first test run of a given project:
>
>    test1: Pass
>    test2: Pass
>    test3: Fail
>
> Then, consider the following partial test results for a second test run
> of the same project:
>
>    test1: Pass
>
> The second test run only executed the first test and, for the sake of
> this example, we can safely assume that this was deliberate and the
> test runner process did not crash and burn.

>
> My question is: can we make any reasonable assumptions about the tests
> that were not run?

The assumption is that something was changed in the release that QA 
thought could break test1 so only ran it to confirm. The traditional 
assumption for the other cases is that their previous run states have 
not changed as well.

> assume that test results remain the same until proven otherwise.

Exactly

So,
> I'm calling for your opinions on what you consider is reasonable.
>
> Thanks!
>


-- 
Chris Gregan
QA Manager
OEM Services Group
Canonical USA Inc.
cgregan at canonical.com
cgregan[irc.freenode.net]
W-781-761-9448

----
1024/8806032D
E70F 7391 6C78 9B9E 6461 1CC7 B168 E1E7 8806 032D




More information about the Ubuntu-qa mailing list