[apparmor] autopkgtests (DEP-8)

[Christian Boltz]

Hello,

Am Sonntag, 25. Juni 2017, 11:18:39 CEST schrieb intrigeri:
> autopkgtests are…
> 
>   "as-installed" tests of packages, i.e. the testing of packages in
>   a context as close as possible to a Debian system where the packages
> subject to testing are properly installed.

For a nice definition of "as close to possible to a real system", have a 
look at openQA [1]. It does everything from installing or upgrading the 
distribution up to things like opening websites in firefox, clicking 
buttons, comparing screenshots with the actual screen content etc.

BTW: Fedora also started to use openQA after some SUSE people started 
testing Fedora "just for fun" and found quite some bugs ;-)

If you are crazy enough to want this, openQA can even test Windows ;-)

> Yesterday I've added some autopkgtests to the apparmor Debian package:
> 
>  
> https://alioth.debian.org/scm/loggerhead/collab-maint/apparmor/revisi
> on/1630
> 
> For now I'm running tests (with USE_SYSTEM=1) in these directories:
> binutils, parser.

You can/should also run the utils tests ;-)

> Distro maintainers, are you doing similar things?
> 
> I'm particularly interested in:
> 
>  * examples of how to run more tests against the installed version,
>    ideally outside of a source tree that was built already;

Not exactly what you asked for, but it might still be interesting:

For the openSUSE package, I run the checks in-tree (without USE_SYSTEM).


%check
%if %{with python3}
export PYTHON=/usr/bin/python3
export PYTHON_VERSIONS=python3
%endif

make check -C libraries/libapparmor
make check -C parser
make check -C binutils

# profiles make check fails for the utils (libapparmor PYTHONPATH
# issues), therefore only do parser-based checks
# also, check-parser breaks if using 'make -C' (but works if cd'ing into 
# the directory)
(cd profiles && make check-parser)

make check -C utils


BTW: If someone has an idea about the strange make -C issue for the 
profiles directory - patches welcome ;-) Same for the PYTHONPATH issue.


Maybe packaging all the tests into an apparmor-tests package, and then 
building a dummy package that runs the tests (without having the source 
tree available) would be an option. However, I'm not sure if the gain is 
worth this effort.


I also asked to get some tests included in openQA (which would mean to 
have them run each time a new Tumbleweed snapshot gets released - but 
unfortunately the openQA team is too busy, so I'm still on the waiting 
list to get this done :-(

The tests I want in openQA are more user-centric:


function errorexit() {
        echo "*** $1 ***" >&2
        exit 1
}

export LC_ALL=C  # make sure all messages are in english, because we grep for some of them


aa-status --enabled || errorexit "AppArmor not enabled"

test `aa-status --enforced` -gt 0 || errorexit "no profiles loaded in enforce mode"

# make sure all running processes have their profile applied
aa-status  | grep '^0 processes are unconfined but have a profile defined.' || { aa-status; errorexit "unconfined processes with a profile defined"; }

# test ping profile
ping -c1 127.0.0.1 || errorexit "ping IPv4 failure"
# Leap 42.x has an older IPv4-only ping (and a separate ping6 command, which we don't test here)
# ping in Tumbleweed can ping IPv6 adresses
test $(rpm --eval '%suse_version') -gt 1320 && { ping -c1 '::1' || errorexit "ping IPv6 failure"; }

# test if mktemp works
tempfile=$(mktemp) || errorexit "mktemp failed (without AppArmor restrictions)"
rm -f "$tempfile"
# provoke a failure by using mktemp with the klogd profile (which isn't allowed to write in /tmp)
mktempmsg=$(aa-exec -p klogd mktemp 2>&1)
test $? == 1 || errorexit "mktemp did not fail with the klogd profile"
echo "$mktempmsg" | grep 'Permission denied' || { echo "$mktempmsg"; errorexit "mktemp didn't fail with 'Permission denied'"; }

# make sure aa-complain works and really sets the profile into complain mode
aa-complain /etc/apparmor.d/sbin.klogd || errorexit "aa-complain failed"
# this will also allow mktemp to create a tempfile
tempfile=$(aa-exec -p klogd mktemp 2>&1) || errorexit "mktemp with complain mode klogd profile failed"
rm -f "$tempfile"

# switch profile back to enforce mode
aa-enforce /etc/apparmor.d/sbin.klogd || errorexit "aa-enforce failed"

systemctl status apparmor.service || errorexit "systemctl status failed"
systemctl is-enabled apparmor.service || errorexit "systemctl is-enabled failed"

echo
echo "AppArmor tests successful :-)"


>  * whether Ubuntu has other autopkgtests, and where they are defined.

Ubuntu runs lots of AppArmor tests as part of
https://launchpad.net/qa-regression-testing/

I played with these tests (only the AppArmor-related ones) on openSUSE a 
while ago. IIRC I found two generic problems that prevented them from 
"simply running":
- Ubuntu creates a group for each user, while openSUSE doesn't do this 
  and has a "users" group instead. This of course results in different 
  behaviour for the pam_apparmor tests.
- tests for dbus etc. won't work - this should be fixed as soon as all
  the kernel patches got upstreamed.


Regards,

Christian Boltz

[1] https://openqa.opensuse.org/ and http://open.qa
-- 
a computer without an Internet connection is essentially a very
expensive DVD player
[http://www.randsinrepose.com/archives/2006/07/10/a_nerd_in_a_cave.html]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20170625/fb732468/attachment.pgp>