[merge] add TestNotApplicable and test documentation

[Martin Pool]

On 8/23/07, Aaron Bentley <aaron.bentley at utoronto.ca> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Pool wrote:
> > I want to add some per-inventory-implementation tests.  I've added
> > some documentation of the per-implementation test framework, and also
> > a distinct exception for 'test not applicable'.
>
> That part of this submission seems good.

Thanks.

>
> > +======================= ======= ======= ========
> > +result                  strict  default lax
> > +======================= ======= ======= ========
> > +TestSkipped             pass    pass    pass
> > +TestNotApplicable       pass    pass    pass
> > +TestPlatformLimit       pass    pass    pass
> > +TestDependencyMissing   fail    pass    pass
> > +KnownFailure            fail    fail    pass
> > +======================= ======= ======= ========
>
> I don't like this.  To me, the point of KnownFailure is to enable us to
> merge test cases for bugs that haven't been fixed yet.  My approach is
> that it's better to have a test case for a bug we haven't fixed than to
> have nothing.

I agree.  I wrote this part before --strict was added, and what was
actually committed makes it consider KnownFailure an overall failure,
and that seems better to me.  I'll change that to 'fail pass pass' if
that's ok with you?

Or maybe it should be 'pass pass pass', if we expect some xfails to be
around for a long time until they're fixed?

-- 
Martin