Questions regarding partial policy load, and the future

[John Johansen]

On 06/19/2010 03:42 AM, Seth Arnold wrote:
>>>> So the question becomes how important is being able to load profiles
>>>> individually?  How often are individual profile reloads used instead of
>>>> just reloading all of policy through the init scripts?
> 
> Creating individual profiles also involves repeatedly reloading
> profile subsets. If the caching is strong enough to allow quickly
> reloading everything when only a single profile changes, that'd be
> alright, but it does not feel fast to me now. (In fact, I use
> apparmor_parser --replace manually when writing profiles, because
> /etc/init.d/apparmor reload is much slower than I'd like.)
> 
Creating profiles also isn't something that is done, frequently by most
people so it can be a little slower than a regular load.  But I do
agree that we don't want it to be too slow.  Right now we just don't
know how fast the cache and merge approach will be.  It will depend
on just what is cached, and also what is getting replaced.  One
possibility is to have multiple levels of cache with 1st level being
individual file parts, the next level certain groups combined, and
the 3rd level being a full policy.

> Perhaps a two-set approach would make sense? One pass for everything
> in enforce mode, one pass for everything in complain mode. That way,
> there are fewer profiles to recompile when developing profiles.
> 
This is a possibility, as I am currently leaning towards having namespaces
have separate policy loads as well.

>>> Right now, all the time.  The idea of on-demand loading for only things
>>> that are about to start up is how it's split up now to spread out the
>>> potential load times.
>>>
>> Right, at the moment we have split load, where we load the early set
>> and then, reload the whole set later.
> 
> I'm seriously confused about this.
> 
> How much faster does the system boot when it loads two sets of
> policies, rather than one policy set?
> 
> How much faster does the system boot when policy loads are 'spread out'?
> 
If everything is cached then not much.  Kees correct me if I am wrong
but the split came about because of serialization, at the early load
point other tasks are blocked waiting on AA to load policy (this is
pre network bringup).

Later on we were doing a parallel profile load with multiple parsers
running.  This got backed out for the moment as the multiple parsers
were bringing several machines to their knees, actually slowing things
down.

> (I'd just assume that filesystem and CPU caches would both be more
> effective with one load up front, done all at once. Keeping caches hot
> and all that.)
> 
Generally yes, very early load can be an exception in that its done before
most services have started, so it is serial to the rest of the system
startup, compared to later when (for Ubuntu at least) the start of some
services can be done in parallel with policy load.

> Are there any potential speedups possible by compiling the #included
> chunks into their own DFAs and caching those? That way, when arekm's
> apache profile loads a given #included piece 1000+ times, it is only
Yes there is some win in doing this.

> compiled once, and the hats need only integrate what is unique and new
> in the hat with the pre-compiled included piece. Would the #included
> chunks be suitable explosion check valves to NFAs? (Probably
> non-optimal, but finding optimal choke points sounds difficult to me.
> :)
> 
Generally no, as finding those points, which don't have to be optimal
just good enough, will depend on the interaction of all rules.