[apparmor] [patch] parser language tests: force using a features file

[John Johansen]

On 04/24/2014 12:27 PM, Steve Beattie wrote:
> On Thu, Apr 24, 2014 at 12:06:14PM -0700, Seth Arnold wrote:
>> On Thu, Apr 24, 2014 at 12:09:42AM -0700, Steve Beattie wrote:
>>> A simple python script is included that was used to generate the
>>> features file based on the current feature set. I'm not sure how to
>>> keep it up to date in an automated fashion as we add more supported
>>> features, however. (make check can't just fail on the features
>>> directory being different; we want builds and tests to run successfully
>>> on older releases where the kernel may not support mediating all the
>>> features we include.)
>>
>> This looks like a useful start but I wouldn't be surprised if there's more
>> pain to be had before we find a solution that makes sense for the
>> "optionally supported" mediations.
> 
> I agree, more pain here is probably coming.
> 
oh a lot more pain by the time I am done with it :)

>> The first thing that comes to mind is maintaining two piles of
>> "expected" outcomes for each feature, but I really don't care for
>> that much.
> 
so this is going to be a necessity, and it will actually be more than 2 out
comes. Once the policy versioning hits it will be the combination of
  policy version for feature (yes this is really needed)
  kernel abi support
  parser support for feature

we can probably knock out the parser support for feature, as the tests
get updated with the parser but that still at least 4 combinations we need
to test and have expected out comes for

> Interestingly, with the current set of committed language tests (I have
> more forthcoming), only one ptrace test failed (ptrace/bad_10.sd). This
yes it is interesting, and a little disappointing our test coverage is
not thorough enough to have broken more

> is because most of the parsing occurs before determining what mediation
> features are supported by the kernel; only stuff like regex conversions
> (and the error checks that go with them) occur after the features
> check. The parser is (currently) almost entirely static with regards
> to what language elements it will accept as input.
right this will change at some point. The basic plan for the parser is
to make it a hybrid of recursive decent and lalr.  We are going to
start passing the block context down to the parsing as soon as I can
get those patches finished. This will give us context to handle some
things within the parse better.

Also its required to pick-up the user defined rule templates I keep
babbling about. Which are how I am planning to support new features
in policy on old parsers. Basically a way of adding rule patterns
for I know this rule exists and it looks like this so you can ignore it
if you don't understand it.

> 
> That said, less parser code is being exercised by the language tests
> if when features are detected as not being supported, as some of the
> post processing won't occur. But exercising the post processing code
> completely is not the primary intent of the language tests; it's a
> better target for the regression test suite.
> 
yep