[Bug 1462139] Re: curtain fails with "Device is busy" error during unmount
Scott Moser
smoser at ubuntu.com
Mon Jun 22 16:12:18 UTC 2015
** Description changed:
+ === Begin SRU Template ===
+ [Description]
+ A race condition can occur when invoking grub-install inside the target environment.
+ The failure case shows logs like this:
+ | Installing for i386-pc platform.
+ | Installation finished. No error reported.
+ | umount: /tmp/tmpM4R1dI/target/dev: device is busy.
+ | (In some cases useful info about processes that use
+ | the device is found by lsof(8) or fuser(1))
+ | Unexpected error while running command.
+ | Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
+ | Exit code: 1
+ | Reason: -
+ | Stdout: ''
+ | Stderr: ''
+ | Installation failed with exception: Unexpected error while running command.
+ | Command: ['curtin', 'curthooks']
+
+ This is believed to be because some process (likely spawned by udev) has
+ open filehandles on /dev when curtin went to clean up the target mounts.
+
+ The solution is to run 'udevadm settle' before unmounting '/dev/' from
+ the target.
+
+ [Impact]
+ The impact is transient failure to install. This race condition is very rarely seen on hardware, but was somewhat easily reproduced in a heavily loaded vmware environment.
+
+ [Test Case]
+ In the original bug-opener's environment it fails fairly reliably under heavy host load using vmware. He would do a deploy to several guests on the same host at the same time and this would reproduce. Unfortunately I was unable to come up with a test case in a less complex environment.
+
+ [Regression Potential]
+ Regression potential should be very low here. The most likely fallout is just additional time for the install as a result of running 'udevadm settle'. A system that did not exhibit this bug will install a small fraction of a second slower.
+
+ $ sudo bash -c 'time for x in "$@"; do udevadm settle; done' -- $(seq 1 100)
+ real 0m0.214s
+ user 0m0.012s
+ sys 0m0.008s
+
+ As shown above, that is likely to be on the order of 1/100th of a
+ second.
+
+ === End SRU Template ===
+
+
Here's the relevant part of the curtin output:
Installing for i386-pc platform.
Installation finished. No error reported.
umount: /tmp/tmpM4R1dI/target/dev: device is busy.
- (In some cases useful info about processes that use
- the device is found by lsof(8) or fuser(1))
+ (In some cases useful info about processes that use
+ the device is found by lsof(8) or fuser(1))
Unexpected error while running command.
Command: ['umount', '/tmp/tmpM4R1dI/target/dev']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
I spoke with Scott Moser on IRC about this, and he suggested the
attached patch, which fixed the problem 100%.
I checked the installation output of a few of my MAAS nodes, and didn't
see any lsof output. So I assume the "udevadm settle" command is the
fix.
(Thanks Scott! I hope we can land this in time for MAAS 1.8.)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to curtin in Ubuntu.
https://bugs.launchpad.net/bugs/1462139
Title:
curtain fails with "Device is busy" error during unmount
To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1462139/+subscriptions
More information about the Ubuntu-server-bugs
mailing list