[apparmor] Fun with mod_apparmor + keepalive + iOS

[Walter Hop]

Hi all,

In our short IRC discussion I promised to give a better reproduction of this issue.

To recap, with Apache + mod_apparmor + Keepalive enabled, some (iOS) clients are triggering an AppArmor violation like the following, causing the HTTP connection to be reset:

  type=1400 audit(1429734615.495:288): apparmor="DENIED" operation="file_perm"
  profile="/usr/sbin/apache2//HANDLING_UNTRUSTED_INPUT"
  name="/sites/keepalive/www/wp-content/plugins/wptouch/resources/icons/elegant/Paper.png"
  pid=17735 comm="apache2" requested_mask="r" denied_mask="r" fsuid=33 ouid=1000

I made packet dumps when triggering the problem with an iOS device, and from those, distilled a sequence of GET requests and the time difference between them: http://lf.ms/requests.json

I made a small script to replay those requests with exact timings to a local VM. This reproduces the problem nearly 100% for me. The script is here: http://lf.ms/replay.phps

# php replay.php
Sleeping 0 usec... Getting /
Sleeping 31012 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/css/wptouch-icons.css?ver=2.3.3
Sleeping 354 usec... Getting /wp-content/plugins/quicktime-embed/qtobject.js
Sleeping 478 usec... Getting /wp-content/wptouch-data/cache/wptouch-567c11e69e00b40b5814ded9a6320ea84048adc2.js
Sleeping 6 usec... Getting /wp-content/plugins/wp-disable-comments/javascript/wp-disable-comments.js?ver=0.4
Sleeping 1078 usec... Getting /wp-content/plugins/wptouch/resources/icons/elegant/Paper.png
Sleeping 9755 usec... Getting /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1
Sleeping 3798 usec... Getting /wp-content/plugins/wptouch/themes/foundation/default/style.css?ver=3.7.5.3
Sleeping 25 usec... Getting /wp-content/plugins/wptouch/themes/bauhaus/default/style.css?ver=3.7.5.3
Sleeping 9679 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/pushit/pushit.css?ver=4.1.1
Sleeping 6477 usec... Getting /wp-includes/js/jquery/jquery.js?ver=1.11.1
Sleeping 166401 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/font/wptouch-icons.woff?64777116
Notice: fwrite(): send of 180 bytes failed with errno=104 Connection reset by peer in /home/walter/replay.php on line 18
Yay! Connection was broken!

As you can see, the server closes the connection prematurely. At this moment the ^HANDLING_UNTRUSTED_INPUT audit log for "Paper.png" has happened.

Funny enough, with this example, I always get the error in "Paper.png"! That file is very small by the way (336 bytes). Any GET requests coming after "Paper.png" are not found in Apache's access_log (although doing the extra requests does seem necessary for the problem to happen!) Nothing in Apache’s error_logs.

When I omit stuff from requests.json or shuffle it around, I don't get the problem that easily. The problem likely depends on very specific timing. But, the good thing is that the test case triggers the problem on multiple Ubuntu 14.04 LTS machines.

Various observations:
1) "EnableSendfile Off" reduces the problem 75%. My test case stops reproducing, but production still has some failures for iOS users.
2) "EnableMMAP Off” affects timing, sometimes making the error switch between other requested files.
3) "KeepAliveTimeout" is unrelated, the problem happens instantly into the connection, so the problem is not related to normal connection teardown.

What is a good next step?

The packet capture is probably not generally useful, but there might be a chance to reproduce the problem on a different machine; I can pack up the essential files from my web root to go with my requests.json if needed.

Since I've got an easy way to reproduce on a test VM, I could also experiment with debug mod_apparmor builds or different kernels, although a bit of guidance in getting these running would be helpful (my experience is mostly with FreeBSD). I run Ubuntu 14.04 LTS, apache24+php56 from ondrej's PPA, and compiled mod_apparmor from source. Trying the apparmor userland backports screwed up my system and didn’t seem to help, but I could give it another try on the VM.

Cheers!
WH

-- 
Walter Hop | PGP key: https://lifeforms.nl/pgp

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20150423/56481f54/attachment.html>