[Bug 1338637] Re: continuous re-exec can result in a build-up of inotify fds [SRU]

[Steve Langasek]

The upstream fix for this was included in release 1.13, so this bug is
fixed for utopic and later.

** Changed in: upstart (Ubuntu)
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to upstart in Ubuntu.
https://bugs.launchpad.net/bugs/1338637

Title:
  continuous re-exec can result in a build-up of inotify fds [SRU]

Status in upstart :
  Fix Released
Status in upstart package in Ubuntu:
  Fix Released
Status in upstart source package in Trusty:
  Confirmed

Bug description:
  = PROBLEM =

  The version of Upstart currently in trusty (1.12.1-0ubuntu4.2) suffers
  from a couple of problems which in combination could make upgrading
  difficult on systems with high uptimes.

  The 2 issues are:

  == bug 901038 ==

  The issue here is that telinit in trusty is not fully synchronous. This
  is not normally a problem. However if Upstart is re-exec'ed using
  'telinit u', as is done by the maintainer scripts for the following packages...

    - libc6
    - libdbus-1-3
    - libjson-c2
    - libnih1
    - libnih-dbus1
    - libselinux1
    - libsepol1
    - upstart

  ... and if that operation is slow (it is normally extremely fast),
  subsequent package upgrades (part of the same apt-get run) which *also*
  need to restart Upstart may fail since 'telinit u' is unable to connect
  to PID 1 as a result of Upstart still being in the process of
  re-exec'ing from the last call to 'telinit u'.

  == bug 1338637 ==

  This bug only really affects server systems which have high uptimes. If
  a re-exec is triggered via 'telinit u' as part of a package upgrade (see
  above), Upstart will consume 2 additional inotify watches. Although this
  doesn't affect the correct behaviour of Upstart, it does have two
  repercussions:

  a) It wastes inotify watches.
  b) It slows down an Upstart re-exec ('telinit u').

  Note that the slow-down for a vanilla Trusty server may not be
  detectable unless the value of /proc/sys/fs/inotify/max_user_instances
  has been raised above the default value of 128.

  = FIX DETAILS =

  == bug 901038 ==

  The fix for this bug now makes 'telinit u' block until the re-exec
  operation has completed fully.

  Technically, it is not possible to make 'telinit u' synchronous since
  because D-Bus connections cannot be serialised and since the 'telinit u'
  request is made via a D-Bus connection, when Upstart re-exec's, it has
  to sever all D-Bus connections, including the 'telinit u' D-Bus
  connection. As such, 'telinit u' now performs the following operations:

    - Requests synchronously that Upstart re-exec itself.
    - Polls Upstart "forever" by attempting to connect to PID 1 and if
      that operation fails, waiting for a period, the retrying.

  The code is well commented to explain this less-than-ideal but
  nonetheless essential poll operation:

    http://bazaar.launchpad.net/~upstart-
  devel/upstart/trunk/view/head:/util/telinit.c#L171

  == bug 1338637 ==

  This was a simple bug fix.

  = FIX AVAILABILITY =

  TBD.

  = IMPACT =

  On a high uptime server, the more times that 'telinit u' is called (as
  the result of normal apt-get updates to any of the packages listed under
  bug 901038 above), the slower the operation will take to complete, and
  thus the likelihood of bug 901038 being seen will increase.

  = JUSTIFICATION =

  System updates need to "just work". However, as outlined above, the
  longer a systems uptime, the more likely it is to be affected by this
  issue which increases the likelihood of a system update failure.

  This issue needs to be fixed as soon as possible to minimise issues for
  trusty users and administrators, particularly before any potential
  upgrade to Utopic.

  = TEST CASE =

  == To demonstrate bug 1338637 ==

  $ sudo telinit u
  $ sudo ls -l /proc/1/fd|grep inotify|wc -l

  The correct output from the above 2 commands should be simply "2".

  However, on trusty systems, if the commands above are run after a fresh
  boot, the value will in all likelihood be "4". Also, every subsequent
  'telinit u' will increase the number displayed by 2.

  == To demonstrate bug 901038 ==

  for i in $(seq 10); do telinit u; done

  There should be no output from the command-line above, but on a trusty
  system the chances are that one or more lines will display like this:

  $ for i in $(seq 5); do sudo telinit u;done
  telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
  telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
  telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused

  == To prove the overall problem has been fixed ==

  1) Download the attached "force-reexec.sh" onto a trusty system.
  2) Make executable:

     $ chmod 755 ./force-reexec.sh

  3) Run script as root, specifying a number of iterations (100) and
     saving output to file "./typescript.bad":

     $ script -c 'sudo ./force-reexec.sh 100' typescript.bad

  4) Upgrade to latest version of Upstart that fixes the 2 bugs.

  5) Reboot.

  6) Re-run the script

     $ script -c 'sudo ./force-reexec.sh 100' typescript.good

  7) Check the results:

     - file "typescript.bad" will show an increasing number of watches
       and *MAY* show a slowly increasing restart time (depending on
       the value of /proc/sys/fs/inotify/max_user_instances (see above)).

    - file "typescript.good" should consistently show 2 watches and a
      restart time of 0 on average.

  = TEST RESULTS =

  See the 2 attached typescript files showing a run of force-reexec.sh
  on an affected system, and the other showing the output of force-
  reexec.sh on a system where the problem has been resolved.

  [REGRESSION POTENTIAL]

  The only theoretically potential issue is what happens if the continuous
  poll performed by 'telinit u' never completes. However, this should
  never happen since:

  1) Upstart checks to ensure that it can serialise its own state *before*
     it actually performs the re-exec. If it cannot for some reason (the
     only possibility here is critically low memory), it will
     automatically degrade to a stateless re-exec. A stateless re-exec can
     only fail if the on-disk "/sbin/init" binary or associated libraries
     are somehow corrupted.

  2) If, after checking that its own state can be serialised, the actual
     re-exec operation fails "mid-flight", again, Upstart will
     automatically revert to performing a simple stateless re-exec which
     can only fail if the on-disk "/sbin/init" binary or associated
     libraries are somehow corrupted.

  
  = OTHER INFORMATION =

  == Upgrade Procedure ==

  Note that updating a system to the latest version of Upstart that fixes
  the two bugs outlined above will stop the number of inotify watches
  growing, but will NOT bring the value down to the expected "2".

  As such, to correct the problem fully, it is necessary to reboot the
  system after successfully upgrading the Upstart package.

  == D-Bus Daemon ==

  Note that although Upstart makes heavy use of D-Bus, it does not require
  a D-Bus daemon to be running. Specifically, 'telinit u' communicates
  with PID 1 via a private abstract D-Bus socket, so is immune from issues
  with dbus-daemon(1).

  == Work Around ==

  If your system is affected by this issue, you can perform the
  following actions to work around the issue in a non-invasive manner by
  simply disabling all calls to 'telinit u' via package maintainer
  scripts):

  $ sudo mkdir /root/bin
  $ sudo ln -s /bin/true /root/bin/telinit
  $ sudo su -
  # (PATH=/root/bin:$PATH which telinit)
  # (PATH=/root/bin:$PATH apt-get update && apt-get upgrade)
  # exit
  $ sudo /sbin/telinit u

  Note the final required call to the real telinit to perform a real
  Upstart restart *after* apt-get has completed.

  
  = Original Description =

  $ sudo ls -al /proc/1/fd|grep anon|wc -l
  2

  $ i=0; while [ $i -lt 1024 ]; do sudo telinit u; i=$((i+1)); done

  $ sudo ls -al /proc/1/fd|grep anon|wc -l
  106

To manage notifications about this bug go to:
https://bugs.launchpad.net/upstart/+bug/1338637/+subscriptions