[apparmor] Fun with mod_apparmor + keepalive + iOS

Wed Apr 22 23:46:34 UTC 2015

Thanks, this is excellent work.

On Thu, Apr 23, 2015 at 12:10:21AM +0200, Walter Hop wrote:
> In our short IRC discussion I promised to give a better reproduction of this issue.
> 
> To recap, with Apache + mod_apparmor + Keepalive enabled, some (iOS) clients are triggering an AppArmor violation like the following, causing the HTTP connection to be reset:
> 
>   type=1400 audit(1429734615.495:288): apparmor="DENIED" operation="file_perm"
>   profile="/usr/sbin/apache2//HANDLING_UNTRUSTED_INPUT"
>   name="/sites/keepalive/www/wp-content/plugins/wptouch/resources/icons/elegant/Paper.png"
>   pid=17735 comm="apache2" requested_mask="r" denied_mask="r" fsuid=33 ouid=1000
> 
> I made packet dumps when triggering the problem with an iOS device, and from those, distilled a sequence of GET requests and the time difference between them: http://lf.ms/requests.json
> 
> I made a small script to replay those requests with exact timings to a local VM. This reproduces the problem nearly 100% for me. The script is here: http://lf.ms/replay.phps

I also am unable to see this script, as a mod_security firewall(?) seems
to block it.

> # php replay.php
> Sleeping 0 usec... Getting /
> Sleeping 31012 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/css/wptouch-icons.css?ver=2.3.3
> Sleeping 354 usec... Getting /wp-content/plugins/quicktime-embed/qtobject.js
> Sleeping 478 usec... Getting /wp-content/wptouch-data/cache/wptouch-567c11e69e00b40b5814ded9a6320ea84048adc2.js
> Sleeping 6 usec... Getting /wp-content/plugins/wp-disable-comments/javascript/wp-disable-comments.js?ver=0.4
> Sleeping 1078 usec... Getting /wp-content/plugins/wptouch/resources/icons/elegant/Paper.png
> Sleeping 9755 usec... Getting /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1
> Sleeping 3798 usec... Getting /wp-content/plugins/wptouch/themes/foundation/default/style.css?ver=3.7.5.3
> Sleeping 25 usec... Getting /wp-content/plugins/wptouch/themes/bauhaus/default/style.css?ver=3.7.5.3
> Sleeping 9679 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/pushit/pushit.css?ver=4.1.1
> Sleeping 6477 usec... Getting /wp-includes/js/jquery/jquery.js?ver=1.11.1
> Sleeping 166401 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/font/wptouch-icons.woff?64777116
> Notice: fwrite(): send of 180 bytes failed with errno=104 Connection reset by peer in /home/walter/replay.php on line 18
> Yay! Connection was broken!
> 
> As you can see, the server closes the connection prematurely. At
> this moment the ^HANDLING_UNTRUSTED_INPUT audit log for "Paper.png"
> has happened.
>
> Funny enough, with this example, I always get the error in
> "Paper.png"! That file is very small by the way (336 bytes). Any GET
> requests coming after "Paper.png" are not found in Apache's access_log
> (although doing the extra requests does seem necessary for the problem
> to happen!) Nothing in Apache’s error_logs.
>
> When I omit stuff from requests.json or shuffle it around, I don't get
> the problem that easily. The problem likely depends on very specific
> timing. But, the good thing is that the test case triggers the problem
> on multiple Ubuntu 14.04 LTS machines.
>
> Various observations: 1) "EnableSendfile Off" reduces the problem
> 75%. My test case stops reproducing, but production still has some
> failures for iOS users.  2) "EnableMMAP Off” affects timing,
> sometimes making the error switch between other requested files.  3)
> "KeepAliveTimeout" is unrelated, the problem happens instantly into the
> connection, so the problem is not related to normal connection teardown.
>
> What is a good next step?
>
> The packet capture is probably not generally useful, but there might be
> a chance to reproduce the problem on a different machine; I can pack
> up the essential files from my web root to go with my requests.json
> if needed.
>
> Since I've got an easy way to reproduce on a test VM, I could also
> experiment with debug mod_apparmor builds or different kernels,
> although a bit of guidance in getting these running would be helpful
> (my experience is mostly with FreeBSD). I run Ubuntu 14.04 LTS,
> apache24+php56 from ondrej's PPA, and compiled mod_apparmor from
> source. Trying the apparmor userland backports screwed up my system
> and didn’t seem to help, but I could give it another try on the VM.

Running mod_apparmor compiled from source is fine. One thing you can try
to do is to reproduce the issue with debug logging enabled for
mod_apparmor. You would add the following like so in your apache config:

  LogLevel apparmor:debug

http://httpd.apache.org/docs/current/logs.html#permodule documents the
feature. That said, the additional logging load may mess with the timing
involved if the issue is the result of a race. And obviously, you
wouldn't want to enable this in a production environment.

Thanks again for digging into this.

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20150422/d78668ea/attachment-0001.pgp>