Interpreting test results across test runs

Zygmunt Krynicki zygmunt.krynicki at canonical.com
Thu Nov 11 23:02:10 UTC 2010


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

W dniu 03.11.2010 17:22, Marc Tardif pisze:
> Hi folks,
> 
> I would appreciate your feedback about interpreting test results across
> multiple test runs. For example, consider the following test results for
> the first test run of a given project:
> 
>   test1: Pass
>   test2: Pass
>   test3: Fail
> 
> Then, consider the following partial test results for a second test run
> of the same project:
> 
>   test1: Pass
> 
> The second test run only executed the first test and, for the sake of
> this example, we can safely assume that this was deliberate and the
> test runner process did not crash and burn.
> 
> My question is: can we make any reasonable assumptions about the tests
> that were not run? This can be a matter of opinion where one extreme
> might not make any assumptions at all, whereas another extreme might
> assume that test results remain the same until proven otherwise. So,
> I'm calling for your opinions on what you consider is reasonable.

Hi Marc

Unless you have reliable information on how to handle such condition in
the test case meta data *and* can sufficiently guarantee that the
meda-data is accurate and up-to-date then you should do very little more
than notify the user that the particular test was not run (or not
present in the test result data, I don't know how you handle that part).

Perhaps if you can explain the kind of analysis you are doing then we
can come up with some reasonable set of guidelines for such tests.

Thanks
ZK
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJM3HXxAAoJEKkR4hQBRI+l9sQH/iF0VD48RO35qCiLny0fHa+p
jLnmfYaiEm5eTRYYVRf7Tkhen/cPH0zF0Fd6PqP/IfSQikxKMPnqRYGDM8bW+Qms
DcNuEzgVNEj4fdVOTGfVbodFGcEmb4gf9Bw7pKkv5XUo068/+cs9W1TITeUOr78T
2BPVk7AHTtzC3OpSVO3KDpwP5Gf4d/PBIBot44BfVZC3IExAmg/JtiWjA/TY/Pey
S5RJag0l4GYGBh5H96gkxHa90HfES5gF7i0OeLG9puNWW/PbF7SpDKIAtLO8ZJW/
9J9A4m2pTPSWdBGpeoFmB+5/Sznucgrv3R2y+3o3UpGYbAp1NC668RmuPnaZE7E=
=Wswr
-----END PGP SIGNATURE-----




More information about the Ubuntu-qa mailing list