[Bug 686832] Re: multipath-tools-boot, root mount failed - Device or resource busy

[Serge Hallyn]

** Description changed:

+ SRU Justification:
+ 1. Impact: servers with multipath-tools-boot installed cannot mount root on boot
+ 2. How bug addressed: The initramfs local-top script for multipath-tools waits for udev to handle all device renames resulting from multipath starting up before continuing on to mountroot.
+ 3. Patch: is attached as a debdiff
+ 4. To reproduce: install a server with multipath storage controller, install multipath-tools-boot, and reboot.
+ 5. Regression potential: the fix is localized to an initramfs script shipped with multipath-tools-boot, so if there are regressions they should only affect multipath users, who cannot boot without some fix anyway.
+ 
  Binary package hint: multipath-tools-boot
  
  After installing "multipath-tools-boot", initramfs fails to mount /root
  by uuid with error: "failed: Device or resource busy".  And boot process
  falls into (initramfs) prompt.
  
  **Boot/Screen Output:**
  Begin: Running /scripts/local-premount ... done.
  mount: mounting /dev/disk/by-uuid/082f71b9-47dc-4d4b-bdfa-d5ca61f1e045 on /root
  failed: Device or resource busy
  Begin: Running /scripts/local-bottom ... done.
  done.
  <....> other failure messages mounting (dev, sys, proc, etc.) <....>
- (initramfs) 
- 
+ (initramfs)
  
  **Further Debugging & Info:**
  Procedures & commands, run shortly after (initramfs) prompt appears.
  (initramfs) ls -l /dev/disk/by-uuid/
  lrwxrwxrwx    1       10 082f71b9-47dc-4d4b-bdfa-d5ca61f1e045 -> ../../dm1
  [ This appears correct!?! ]
  
  (initramfs) mount /dev/disk/by-uuid/082f71b9-47dc-4d4b-bdfa-d5ca61f1e045 /root
  [ SUCCESS ! ]
- 
  
  **WORKAROUND:**
  I have created an initramfs script to slow-down the root mounting attempt:
  
  File: /usr/share/initramfs-tools/scripts/local-top/waitfor-multipath
  Content:
  #!/bin/sh
  VERBOSITY=0
  PREREQ="multipath"
  prereqs() { echo "$PREREQ"; }
  case $1 in
  prereqs)
- 	prereqs
- 	exit 0
- 	;;
+  prereqs
+  exit 0
+  ;;
  esac
  
  echo "Waiting 5 seconds for multipaths & udev?? to settle down..."
  sleep 5
  exit 0
  <----END---->
  
  My guess for a general/global bug-fix is multipath-tools-boot script to
  include a slight delay/hold-down timer; to allow outside device-mapping
  procedures to complete.  ..or a confirmation that dm devices have
  finished mapping.
  
- 
  **Observations:**
  It appears multipath-tools-boot initramfs script executes successfully VERY quickly, and initramfs scripts continue with attempting to mount the root file system.  Although, dm-# /dev/disk/by-uuid mappings are not instantly ready.  This is why manually typing the same mount command at prompt is immediately successful.
  
  _To duplicate error in a non-boot scenario:_
  This exact same behavior can be observed & duplicated on running servers; when mounting a multipath (SAN) volume.  As example; mounting multipath-SAN targets will create: /dev/sda, /dev/sdb, /dev/sdc, etc....   And multipath-tools+devmapper will then create the shared/linked /dev/dm-# mappings.  You are then only supposed to use the /dev/dm-# virtual devices to access multipathed SAN devices (multipaths.)
  
  .... If you then attempt to mount a direct SAN volume /dev/sd?# an error
  is raised: "mount: /dev/sd?# already mounted or /<mount_point> busy"
  
- 
  **Software & Versions:**
-  OS: Ubuntu 10.10 (Maverick - server)
-  Related Packages & versions:
-   multipath-tools  0.4.8-14ubuntu4
-   multipath-tools-boot  0.4.8-14ubuntu4
-   udev  162-2.1
-   initramfs-tools  0.98.1ubuntu6
-   initramfs-tools-bin  0.98.1ubuntu6
-   open-iscsi  2.0.871-0ubuntu5
-   open-iscsi-utils  2.0.871-0ubuntu5
-   libdevmapper1.02.1  2:1.02.39-1ubuntu6
+  OS: Ubuntu 10.10 (Maverick - server)
+  Related Packages & versions:
+   multipath-tools  0.4.8-14ubuntu4
+   multipath-tools-boot  0.4.8-14ubuntu4
+   udev  162-2.1
+   initramfs-tools  0.98.1ubuntu6
+   initramfs-tools-bin  0.98.1ubuntu6
+   open-iscsi  2.0.871-0ubuntu5
+   open-iscsi-utils  2.0.871-0ubuntu5
+   libdevmapper1.02.1  2:1.02.39-1ubuntu6
  
  **Hardware Details:**
-  Servers affected: HP Proliant DL385 (G5, G5p, G6) (and others I'm sure.)
-  Storage/SAN Type: iSCSI
-  Storage/SAN Hardware: HP StorageWorks MSA2312i (and others I'm sure.)
-  
-  This problem affects servers with iSCSI-boot NICs, AND servers which have a USB-Key with /boot partition.
+  Servers affected: HP Proliant DL385 (G5, G5p, G6) (and others I'm sure.)
+  Storage/SAN Type: iSCSI
+  Storage/SAN Hardware: HP StorageWorks MSA2312i (and others I'm sure.)
  
+  This problem affects servers with iSCSI-boot NICs, AND servers which
+ have a USB-Key with /boot partition.
  
  Please let me know if more information is necessary!

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in ubuntu.
https://bugs.launchpad.net/bugs/686832

Title:
  multipath-tools-boot, root mount failed - Device or resource busy