[apparmor] [PATCH] Add support for variable expansion in profile names, and attachments

[Steve Beattie]

On Tue, Jun 09, 2015 at 05:31:06AM -0700, John Johansen wrote:
> On 06/09/2015 04:45 AM, Steve Beattie wrote:
> > On Mon, Jun 08, 2015 at 08:46:09PM -0700, John Johansen wrote:
> >> err shouldn't an xfail result in the test set failing if it actually fails?
> >> that way we can fix the test to be fail instead of xfail
> > 
> > That's not the way perl's todo mechanism works by default. Though it's
> > likely possible to do some trickiness with Test::Harness to figure out
> > if any todo tests are passing and register the run as "failing".
> > 
> then why would I want to use it, over using the "wrong" EXRESULT + a comment
> about what the test should do?
> 
> using the "wrong" EXRESULT ensures that the test will pop up when the code
> that fixes the xfail, xpass is added. This ensure that the test will get
> updated, otherwise, people forget and the tests don't get updated.

I agree that people forget and the tests don't get updated, but it's
also the case that people go "Oh, the tests are passing, there's
nothing to fix there" if the wrong expected result is recorded and so
the fact that the test isn't failing when it should be gets forgotten
as well. At least with todo() there's a reminder that you have bugs
that need to be fixed if the reported results are ever read (mind you,
that doesn't mean they do or can be easily; see the XFAIL/XPASS tests
in the regression tests for additional examples).

Additionally I can see an argument as to why having a todo test that
passes not cause the test harness to report failure: while we don't
currently have tests that for whatever reason fail sporadically or
only fail on a specific set of architectures, in dealing with other
code bases I have seen that be the case.

That said, for our usage, it would make sense to have a passing todo()
fail 'make check' so that we can correct the state of the test. But
I don't have the time at the moment to dig into Test::Harness to make
the parser sanity test script do that.

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20150609/4c5d4265/attachment.pgp>