[Bug 1160490] Re: race condition updating statefile

[Scott Moser]

Ok. I just confirmed that this change caused bug 1226067.

 * boot fresh raring cloud image (20130914)
 * ifquery works as expected
 * enable saucy-proposed, apt-get update, apt-get install ifupdown
 * reboot
 * ifquery broken
    $ ifquery --list --allow auto
    ifquery: failed to lock lockfile /run/network/.ifstate.lock: Bad file descriptor
   $ dpkg-query --show ifupdown
   ifupdown        0.7.5ubuntu2.1


** Description changed:

  SRU Justification:
- [Impact] 
-  * Users will occasionally see their network interfaces not come up due to race conditions.
+ [Impact]
+  * Users will occasionally see their network interfaces not come up due to race conditions.
  
  [Test Case]
-  * See this comment: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/996369/comments/35
+  * See this comment: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/996369/comments/35
  
- [Regression Potential] 
-  * This fix backports a change from upstream ifupdown. Instead of locking a statefile it locks a lockfile.
+ [Regression Potential]
+  * This fix backports a change from upstream ifupdown. Instead of locking a statefile it locks a lockfile.
  
  --
  
  Ubuntu 12.04.2
  ifupdown 0.7~beta2ubuntu8
  
  Symptom: Every so often, /etc/init/network-interface.conf fails to bring
  up the loopback interface.
  
  > Mar 25 16:39:37 XXXXXXXX kernel: [   28.793922] init: network-
  interface (lo) pre-start process (1079) terminated with status 1
  
  /var/log/upstart/network-interface-lo shows:
  
  > ifup: failed to overwrite statefile /run/network/ifstate: No such file
  or directory
  
  Relevant section of the ifup sources, in update_state():
  
          if (rename (tmpstatefile, statefile)) {
                  fprintf(stderr,
                          "%s: failed to overwrite statefile %s: %s\n",
                          argv0, statefile, strerror(errno));
                  exit (1);
          }
  
  update_state() opens the statefile, gets a F_SETLKW lock on it, opens a
  tmpfile, filters the contents of the statefile into the tmpfile, closes
  the tmpfile, then renames the tempfile over the statefile.
  
  Once the rename() happens in one instance of ifup, any other blocked
  instances are waiting around to lock a file that no longer exists in the
  filesystem.  Overlap enough instances of ifup just right and you have
  them all locking different copies of statefile, which then doesn't
  prevent any of them from rename()ing tmpstatefile out from underneath
  the others, thus causing their own rename()s to fail with ENOENT.
  
  Example:
  
  Process A starts, opens statefile.
  Process A locks statefile.
  Process B starts, opens statefile.
  Process B waits for lock on statefile.
  Process A renames tmpstatefile to statefile and exits.
  Process B acquires lock on *outdated* statefile FILE pointer.
  Process C starts, opens current statefile (written by Process A).
  Process C locks current statefile.
  ** Two ifups now have locks **
  Process B renames tmpstatefile to statefile and exits.
  Process C tries to rename tmpstatefile, fails because tmpstatefile has already been renamed out from under it by Process B.
  
  NOTE:  Since Process B was operating on an outdated statefile, it has
  also stomped over any changes made by Process A, so simply making the
  tmpstatefile process-specific to avoid rename()ing out from under each
  other won't help.
+ 
+ Related bugs:
+  * bug 1226067:  ifquery fails with bad file descriptor

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1160490

Title:
  race condition updating statefile

Status in “ifupdown” package in Ubuntu:
  Fix Released
Status in “ifupdown” source package in Precise:
  Fix Committed
Status in “ifupdown” source package in Quantal:
  Fix Committed
Status in “ifupdown” source package in Raring:
  Fix Committed
Status in “ifupdown” source package in Saucy:
  Fix Released

Bug description:
  SRU Justification:
  [Impact]
   * Users will occasionally see their network interfaces not come up due to race conditions.

  [Test Case]
   * See this comment: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/996369/comments/35

  [Regression Potential]
   * This fix backports a change from upstream ifupdown. Instead of locking a statefile it locks a lockfile.

  --

  Ubuntu 12.04.2
  ifupdown 0.7~beta2ubuntu8

  Symptom: Every so often, /etc/init/network-interface.conf fails to
  bring up the loopback interface.

  > Mar 25 16:39:37 XXXXXXXX kernel: [   28.793922] init: network-
  interface (lo) pre-start process (1079) terminated with status 1

  /var/log/upstart/network-interface-lo shows:

  > ifup: failed to overwrite statefile /run/network/ifstate: No such
  file or directory

  Relevant section of the ifup sources, in update_state():

          if (rename (tmpstatefile, statefile)) {
                  fprintf(stderr,
                          "%s: failed to overwrite statefile %s: %s\n",
                          argv0, statefile, strerror(errno));
                  exit (1);
          }

  update_state() opens the statefile, gets a F_SETLKW lock on it, opens
  a tmpfile, filters the contents of the statefile into the tmpfile,
  closes the tmpfile, then renames the tempfile over the statefile.

  Once the rename() happens in one instance of ifup, any other blocked
  instances are waiting around to lock a file that no longer exists in
  the filesystem.  Overlap enough instances of ifup just right and you
  have them all locking different copies of statefile, which then
  doesn't prevent any of them from rename()ing tmpstatefile out from
  underneath the others, thus causing their own rename()s to fail with
  ENOENT.

  Example:

  Process A starts, opens statefile.
  Process A locks statefile.
  Process B starts, opens statefile.
  Process B waits for lock on statefile.
  Process A renames tmpstatefile to statefile and exits.
  Process B acquires lock on *outdated* statefile FILE pointer.
  Process C starts, opens current statefile (written by Process A).
  Process C locks current statefile.
  ** Two ifups now have locks **
  Process B renames tmpstatefile to statefile and exits.
  Process C tries to rename tmpstatefile, fails because tmpstatefile has already been renamed out from under it by Process B.

  NOTE:  Since Process B was operating on an outdated statefile, it has
  also stomped over any changes made by Process A, so simply making the
  tmpstatefile process-specific to avoid rename()ing out from under each
  other won't help.

  Related bugs:
   * bug 1226067:  ifquery fails with bad file descriptor

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1160490/+subscriptions