<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi all,</div><div class=""><br class=""></div><div class="">In our short IRC discussion I promised to give a better reproduction of this issue.</div><div class=""><br class=""></div><div class="">To recap, with Apache + mod_apparmor + Keepalive enabled, some (iOS) clients are triggering an AppArmor violation like the following, causing the HTTP connection to be reset:</div><div class=""><br class=""></div><div class=""> type=1400 audit(1429734615.495:288): apparmor="DENIED" operation="file_perm"</div><div class=""> profile="/usr/sbin/apache2//HANDLING_UNTRUSTED_INPUT"</div><div class=""> name="/sites/keepalive/www/wp-content/plugins/wptouch/resources/icons/elegant/Paper.png"</div><div class=""> pid=17735 comm="apache2" requested_mask="r" denied_mask="r" fsuid=33 ouid=1000</div><div class=""><br class=""></div><div class="">I made packet dumps when triggering the problem with an iOS device, and from those, distilled a sequence of GET requests and the time difference between them: <a href="http://lf.ms/requests.json" class="">http://lf.ms/requests.json</a></div><div class=""><br class=""></div><div class="">I made a small script to replay those requests with exact timings to a local VM. This reproduces the problem nearly 100% for me. The script is here: <a href="http://lf.ms/replay.phps" class="">http://lf.ms/replay.phps</a></div><div class=""><br class=""></div><div class=""># php replay.php</div><div class="">Sleeping 0 usec... Getting /</div><div class="">Sleeping 31012 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/css/wptouch-icons.css?ver=2.3.3</div><div class="">Sleeping 354 usec... Getting /wp-content/plugins/quicktime-embed/qtobject.js</div><div class="">Sleeping 478 usec... Getting /wp-content/wptouch-data/cache/wptouch-567c11e69e00b40b5814ded9a6320ea84048adc2.js</div><div class="">Sleeping 6 usec... Getting /wp-content/plugins/wp-disable-comments/javascript/wp-disable-comments.js?ver=0.4</div><div class="">Sleeping 1078 usec... Getting /wp-content/plugins/wptouch/resources/icons/elegant/Paper.png</div><div class="">Sleeping 9755 usec... Getting /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1</div><div class="">Sleeping 3798 usec... Getting /wp-content/plugins/wptouch/themes/foundation/default/style.css?ver=3.7.5.3</div><div class="">Sleeping 25 usec... Getting /wp-content/plugins/wptouch/themes/bauhaus/default/style.css?ver=3.7.5.3</div><div class="">Sleeping 9679 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/pushit/pushit.css?ver=4.1.1</div><div class="">Sleeping 6477 usec... Getting /wp-includes/js/jquery/jquery.js?ver=1.11.1</div><div class="">Sleeping 166401 usec... Getting /wp-content/plugins/wptouch/themes/foundation/modules/wptouch-icons/font/wptouch-icons.woff?64777116</div><div class="">Notice: fwrite(): send of 180 bytes failed with errno=104 Connection reset by peer in /home/walter/replay.php on line 18</div><div class="">Yay! Connection was broken!</div><div class=""><br class=""></div><div class="">As you can see, the server closes the connection prematurely. At this moment the ^HANDLING_UNTRUSTED_INPUT audit log for "Paper.png" has happened.</div><div class=""><br class=""></div><div class="">Funny enough, with this example, I always get the error in "Paper.png"! That file is very small by the way (336 bytes). Any GET requests coming after "Paper.png" are not found in Apache's access_log (although doing the extra requests does seem necessary for the problem to happen!) Nothing in Apache’s error_logs.</div><div class=""><br class=""></div><div class="">When I omit stuff from requests.json or shuffle it around, I don't get the problem that easily. The problem likely depends on very specific timing. But, the good thing is that the test case triggers the problem on multiple Ubuntu 14.04 LTS machines.</div><div class=""><br class=""></div><div class=""><div class="">Various observations:</div></div><div class="">1) "EnableSendfile Off" reduces the problem 75%. My test case stops reproducing, but production still has some failures for iOS users.</div><div class="">2) "EnableMMAP Off” affects timing, sometimes making the error switch between other requested files.</div><div class="">3) "KeepAliveTimeout" is unrelated, the problem happens instantly into the connection, so the problem is not related to normal connection teardown.</div><div class=""><br class=""></div><div class="">What is a good next step?</div><div class=""><br class=""></div><div class="">The packet capture is probably not generally useful, but there might be a chance to reproduce the problem on a different machine; I can pack up the essential files from my web root to go with my requests.json if needed.</div><div class=""><br class=""></div><div class="">Since I've got an easy way to reproduce on a test VM, I could also experiment with debug mod_apparmor builds or different kernels, although a bit of guidance in getting these running would be helpful (my experience is mostly with FreeBSD). I run Ubuntu 14.04 LTS, apache24+php56 from ondrej's PPA, and compiled mod_apparmor from source. Trying the apparmor userland backports screwed up my system and didn’t seem to help, but I could give it another try on the VM.</div><div class=""><br class=""></div><div class="">Cheers!</div><div class="">WH</div><div class=""><br class=""></div><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-- <br class="">Walter Hop | PGP key: <a href="https://lifeforms.nl/pgp" class="">https://lifeforms.nl/pgp</a></div>
</div>
<br class=""></body></html>