[merge] add TestNotApplicable and test documentation

[Aaron Bentley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> I want to add some per-inventory-implementation tests.  I've added
> some documentation of the per-implementation test framework, and also
> a distinct exception for 'test not applicable'.

That part of this submission seems good.

> +======================= ======= ======= ========
> +result                  strict  default lax
> +======================= ======= ======= ========
> +TestSkipped             pass    pass    pass
> +TestNotApplicable       pass    pass    pass
> +TestPlatformLimit       pass    pass    pass
> +TestDependencyMissing   fail    pass    pass
> +KnownFailure            fail    fail    pass
> +======================= ======= ======= ========

I don't like this.  To me, the point of KnownFailure is to enable us to
merge test cases for bugs that haven't been fixed yet.  My approach is
that it's better to have a test case for a bug we haven't fixed than to
have nothing.

If the default behavior is to fail when KnownFailure is raised, this
goes against that.  It means that adding test cases for bugs that
haven't been fixed causes pain, and that acts as a disincentive.  It
means the test cases don't get merged, and may bit-rot.

So I really don't want KnownFailures to cause the suite to fail by
default.  There's not much point having them if they do.

bb:abstain

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGzMnj0F+nu1YWqI0RAgoyAJ9rDWt3i2bY/7b2hyeQkisDT5T5MACfVQ02
o/Bo+pu0CgJJT+NVSoTu/eI=
=b9bU
-----END PGP SIGNATURE-----