[Bug 1526984] Re: ISST-LTE: root mpath device unavailable after installation

Fri Feb 5 15:41:23 UTC 2016

Hum, that's kind of weird. You should see disks being picked up by
multipath if they should, even if they are detected *after* multipath
scripts run (because multipathd...). They wouldn't be detected before
multipath runs, because then they would most likely be remapped anyway
(unless something else claims them).

Could you file a new bug for this, including the output of multipath -v4
taken from the initramfs? You may stop the boot process by adding "break
=pre-multipath" to break boot just before the multipath scripts. At that
point, you'll need to modprobe dm_multipath;  and then you should be
able to run multipath -v4. If you exit from there boot should complete
(hopefully succesfully, since there will have been some extra delay).

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1526984

Title:
  ISST-LTE: root mpath device unavailable after installation

Status in multipath-tools package in Ubuntu:
  Fix Released
Status in multipath-tools source package in Trusty:
  Fix Committed

Bug description:
  [Impact]
  This affects users of multipath systems where devices may be slow to be detected. (See below).

  On some systems disks may be slow to be detected due to a number of
  reasons: slow disks, spin delay, large number of devices to scan, etc.
  If the disk containing the root filesystem fails to be
  detected/scanned when the multipath initramfs scripts are run, the
  system will give up and fail to load the rootfs, which results in a
  failure to boot (and the user will be thrown into a busybox shell in
  the initramfs).

  [Test case]
  Boot the system.

  [Regression Potential]
  Given that this changes the load order for the system, it may cause some delay in booting due to the extra time taken to load all maps for the rootfs to become available through multipath. Also, crashes in multipathd may now affect early boot.

  ---

  Canonical FYI:

  We appear to have a regression in the current 14.04 environment back to
  the behavior originally outlined in Comment #10 of LP Bug 1429327.

  ---Problem---

  A multipath install of 14.04.04 fails to create the install mpath
  device on subsequent boots.  While the root filesystem ends up being
  mounted on top of one of the path devices, other partitions on that
  multipath device will not be automatically found.

  ---Additional details---

  For one example host, the install disk was set up with a root and
  swap partition:

  Dec 14 21:37:55 disk-detect: create: mpath0 (35000c5007655a10f) undef IBM     ,ST300MP0064
  Dec 14 21:37:55 disk-detect: size=279G features='0' hwhandler='0' wp=undef
  Dec 14 21:37:55 disk-detect: |-+- policy='round-robin 0' prio=1 status=undef
  Dec 14 21:37:55 disk-detect: | `- 0:0:7:0 sdc 8:32 undef ready running
  Dec 14 21:37:55 disk-detect: `-+- policy='round-robin 0' prio=1 status=undef
  Dec 14 21:37:55 disk-detect:   `- 1:0:3:0 sdd 8:48 undef ready running

  The resulting fstab entries after install were:

  /dev/mapper/mpath0-part2 /               ext4    errors=remount-ro 0 1
  /dev/mapper/mpath0-part3 none            swap    sw                0 0

  In post-install boots, the multipath command appears to be executed
  prior to any disks being discovered:

  Begin: Loading essential drivers ... done.
  Begin: Running /scripts/init-premount ... done.
  Begin: Mounting root file system ... [    0.612955] ipr: IBM Power RAID SCSI Device Driver version: 2.6.1 (March 12, 2015)
  [    0.612985] ipr 0000:80:00.0: Found IOA with IRQ: 0
  [    0.613478] ipr 0000:80:00.0: ibm,query-pe-dma-windows(53) 800000 8000000 20000164 returned 0
  Begin: Running /scripts/local-top ... [    0.613727] ipr 0000:80:00.0: ibm,create-pe-dma-window(54) 800000 8000000 20000164 10 21 returned 0 (liobn = 0x70000164 starting addr = 8000000 0)
  Begin: Loading multipath modules ... [    0.618790] device-mapper: multipath: version 1.9.0 loaded
  Success: loaded module dm-multipath.
  Failure: failed to load module dm-emc.
  done.
  Begin: Waiting for scsi storage ... done.
  Begin: Discovering multipaths ... [    0.623429] be2net 0002:01:00.0: be2net version is 10.6.0.2
  [    0.623587] be2net 0002:01:00.0: enabling device (0000 -> 0002)
  done.
  [    0.636785] ipr 0000:80:00.0: Using 64-bit direct DMA at offset 800000000000000
  [    0.637343] be2net 0002:01:00.0: PCIe error reporting enabled
  [    0.641435] ipr 0000:80:00.0: Received IRQ : 20
  [    0.641522] ipr 0000:80:00.0: Request for 2 MSIXs succeeded.
  [    0.645713] ipr 0000:80:00.0: Starting IOA initialization sequence.
  [    0.645719] scsi host0: IBM 0 Storage Adapter
  [    0.645741] ipr 0000:80:00.0: Starting IOA initialization sequence.
  [    0.645833] ipr 0001:a0:00.0: Found IOA with IRQ: 0
  [    0.646221] ipr 0000:80:00.0: Adapter firmware version: 13510C00
  [    0.647673] ipr 0000:80:00.0: IOA initialized.
  [    0.648009] ipr 0001:a0:00.0: Received IRQ : 448
  [    0.648041] ipr 0001:a0:00.0: Request for 2 MSIXs succeeded.
  [    0.651764] ipr 0001:a0:00.0: Starting IOA initialization sequence.
  [    0.651769] scsi host1: IBM 0 Storage Adapter
  [    0.651795] ipr 0001:a0:00.0: Starting IOA initialization sequence.
  [    0.652262] ipr 0001:a0:00.0: Adapter firmware version: 13510C00
  [    0.653702] ipr 0001:a0:00.0: IOA initialized.
  [    0.660800] scsi 0:3:0:0: No Device         IBM      57B1001SISIOA    0150 PQ: 0 ANSI: 0
  [    0.660811] scsi 0:3:0:0: Resource path: 0/FE
  [    0.672518] scsi 0:0:0:0: Direct-Access     IBM      ST300MP0064      7D0E PQ: 0 ANSI: 6
  [    0.672530] scsi 0:0:0:0: Resource path: 0/00-0E-08
  ...
  (first device in mpath0 seen at this point):
  [    0.710988] scsi 1:0:3:0: Direct-Access     IBM      ST300MP0064      7D0E PQ
  : 0 ANSI: 6
  [    0.710994] scsi 1:0:3:0: Resource path: 1/00-0E-0A

  The system ends up mounting the root filesystem from /dev/sdc2,
  and the post-mount run of multipath errors out creating mpath0:

  [   10.056286] device-mapper: table: 252:0: multipath: error getting device
  [   10.082644] device-mapper: table: 252:0: multipath: error getting device
  [   10.192614] device-mapper: table: 252:2: multipath: error getting device
  [   10.222609] device-mapper: table: 252:2: multipath: error getting device
  [   10.663458] device-mapper: table: 252:2: multipath: error getting device
  [   10.682690] device-mapper: table: 252:2: multipath: error getting device
  [   10.796743] device-mapper: table: 252:2: multipath: error getting device

  Since no other filesystems depend on mpath0 for this example config,
  the system is able to boot, but lacks swap due to the missing swap
  device.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1526984/+subscriptions