[Bug 1462139] Re: curtain fails with "Device is busy" error during unmount

Scott Moser smoser at ubuntu.com
Mon Jun 22 16:12:18 UTC 2015


** Description changed:

+ === Begin SRU Template ===
+ [Description]
+ A race condition can occur when invoking grub-install inside the target environment.
+ The failure case shows logs like this:
+  | Installing for i386-pc platform.
+  | Installation finished. No error reported.
+  | umount: /tmp/tmpM4R1dI/target/dev: device is busy.
+  |         (In some cases useful info about processes that use
+  |          the device is found by lsof(8) or fuser(1))
+  | Unexpected error while running command.
+  | Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
+  | Exit code: 1
+  | Reason: -
+  | Stdout: ''
+  | Stderr: ''
+  | Installation failed with exception: Unexpected error while running command.
+  | Command: ['curtin', 'curthooks']
+ 
+ This is believed to be because some process (likely spawned by udev) has
+ open filehandles on /dev when curtin went to clean up the target mounts.
+ 
+ The solution is to run 'udevadm settle' before unmounting '/dev/' from
+ the target.
+ 
+ [Impact] 
+ The impact is transient failure to install.  This race condition is very rarely seen on hardware, but was somewhat easily reproduced in a heavily loaded vmware environment.
+ 
+ [Test Case]
+ In the original bug-opener's environment it fails fairly reliably under heavy host load using vmware.  He would do a deploy to several guests on the same host at the same time and this would reproduce.  Unfortunately I was unable to come up with a test case in a less complex environment.
+ 
+ [Regression Potential] 
+ Regression potential should be very low here. The most likely fallout is just additional time for the install as a result of running 'udevadm settle'.  A system that did not exhibit this bug will install a small fraction of a second slower.
+ 
+ $ sudo bash -c 'time for x in "$@"; do udevadm settle; done' -- $(seq 1 100)
+   real  0m0.214s
+   user  0m0.012s
+   sys   0m0.008s
+ 
+ As shown above, that is likely to be on the order of 1/100th of a
+ second.
+ 
+ === End SRU Template ===
+ 
+ 
  Here's the relevant part of the curtin output:
  
  Installing for i386-pc platform.
  Installation finished. No error reported.
  umount: /tmp/tmpM4R1dI/target/dev: device is busy.
-         (In some cases useful info about processes that use
-          the device is found by lsof(8) or fuser(1))
+         (In some cases useful info about processes that use
+          the device is found by lsof(8) or fuser(1))
  Unexpected error while running command.
  Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
  Exit code: 1
  Reason: -
  Stdout: ''
  Stderr: ''
  Installation failed with exception: Unexpected error while running command.
  Command: ['curtin', 'curthooks']
  
  I spoke with Scott Moser on IRC about this, and he suggested the
  attached patch, which fixed the problem 100%.
  
  I checked the installation output of a few of my MAAS nodes, and didn't
  see any lsof output. So I assume the "udevadm settle" command is the
  fix.
  
  (Thanks Scott! I hope we can land this in time for MAAS 1.8.)

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to curtin in Ubuntu.
https://bugs.launchpad.net/bugs/1462139

Title:
  curtain fails with "Device is busy" error during unmount

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1462139/+subscriptions



More information about the Ubuntu-server-bugs mailing list