[apparmor] Fun with mod_apparmor + keepalive + iOS

[Walter Hop]

Hi,

After getting comfortable with mod_apparmor and slowly migrating sites to it, I'm seeing some weird audit logs from the HANDLING_UNTRUSTED_INPUT hat on a virtual host. The weird entries all have this form:

  apparmor="DENIED" operation=“file_perm"
  profile="/usr/sbin/apache2//HANDLING_UNTRUSTED_INPUT"
  name="/usr/opt/httpd/sites/example.com/www/wp-content/themes/art/images/search.gif"
  pid=31631 comm="apache2" requested_mask="r" denied_mask="r" fsuid=33 ouid=1002

These occur every few minutes, although in <1% of requests. (Otherwise the site works totally fine under its AADefaultHatName, which is the only hat that should access sites/example.com/**.)

According to Apache’s access_logs, these resources are actually requested by a client at the exact time of audit entry, and the response appears to be sometimes 0 bytes, so it looks like a reliability issue first and foremost. (It might be a security issue in case clients could force a specific request to stay in HANDLING_UNTRUSTED_INPUT *and* this hat would be more privileged than the uri/vhost-specific hat. Probably a contrived example in most cases.)

After correlating the AppArmor log messages with access_logs, I found that some requirements must be met for the problem to happen:

1. The Apache 2.4 server must have HTTP keep-alive enabled
2. The client user-agent is Safari on iOS (seen 8_1_2, 8_1_3 and 8_2, either iPad or iPhone)
3. The client visits a HTML web page with some sub resources
4. Some requests to sub resources are now liable to cause a file access attempt from HANDLING_UNTRUSTED_INPUT, e.g. one or more gif/jpg/png images.

There are likely additional confounding variables, as I am seeing this problem only on one machine, even though others are running normally for weeks. I suspect it's related to a mostly mobile regional audience and a very narrow timing window during which subsequent HTTP keepalive requests trigger the problem.

Setting "KeepAlive Off" on the server stops the audit entries completely for many hours, so this seems a workaround.

The symptoms seem to be very similar to a thread by Christian Boltz in 2012: https://lists.ubuntu.com/archives/apparmor/2012-March/002414.html (in that case it's .css files, I wonder if it was ever solved?)

From the possible scenarios in that thread, it seems here (due to relationship with keepalive) that either Apache or mod_apparmor would be the most likely location. No errors are logged by Apache (the requests seem to get a 200 response although its bytes transferred are lower than normal requests).

I'd be happy to experiment a bit. I couldn't yet reproduce by talking to port 80, however I can now get a reasonably consistent reproduce by reloading a webpage from my iPhone in a private browsing tab.

Configuration: Ubuntu 14.04 LTS x64, Linux 3.13.0-46-generic, Apache 2.4 prefork 2.4.12-1+deb.sury.org~trusty+5, mod_apparmor 2.9.1.

Cheers!
WH

-- 
Walter Hop | PGP key: https://lifeforms.nl/pgp

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20150321/84a56740/attachment-0001.html>