[Bug 1540401] Re: ISST-LTE: Ubuntu14.04.4 lpar fails to boot after installation: "The disk drive for /boot is not ready yet or not present"
Mauricio Faria de Oliveira
mauricfo at linux.vnet.ibm.com
Thu Feb 11 23:33:29 UTC 2016
Hi @mathieu-tl,
Here is the patch for Xenial.
Differences from the patch for Trusty:
1) install udev rule with priority 56, so it's run after 55-scsi-sg3_id; this way the paths actually have scsi id udev attributes defined by the time 'multipath -c' runs, and now it works fine (fixes the issue you mentioned on IRC).
2) remove the old multipath discovery udev rule (just like you applied for trusty, on multipath-tools 0.4.9-3ubuntu7.5)
3) debian/initramfs/init-top already exists, so just insert the snippet to load the module (rather than create the file).
Test-case:
1) Boot a qemu-kvm guest [1] w/ xenial (LVM on top of multipath) w/ the break=pre-multipath boot option
2) exit all 3 initramfs prompts
3) this should make LVM scan run before multipath discovery; the latter fails to create the devmap, thus mpartX-part2 (for /boot) is not present, and the boot is interrupted.
...
(initramfs) exit
...
(initramfs) exit
...
(initramfs) exit
...
[ 82.361944] device-mapper: table: 252:3: multipath: error getting device
[ 82.362292] device-mapper: table: 252:2: multipath: error getting device
[ 82.399493] device-mapper: table: 252:3: multipath: error getting device
[ OK ] Found device /dev/mapper/mauricfo4--vg-swap_1.
Activating swap /dev/mapper/mauricfo4--vg-swap_1...
[ OK ] Activated swap /dev/mapper/mauricfo4--vg-swap_1.
[ OK ] Reached target Swap.
[ TIME ] Timed out waiting for device dev-mapper-mpatha\x2dpart2.device.
[DEPEND] Dependency failed for /boot.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Clean up any mess left by 0dns-up.
[DEPEND] Dependency failed for File System Check on /dev/mapper/mpatha-part2.
...
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
try again to boot into default mode.
Press Enter for maintenance
(or press Control-D to continue):
root at mauricfo4:~# lvm pvdisplay | grep Name
Found duplicate PV hWoIFGkvc0iVrbZnzhhqxud6QTeotfmQ: using /dev/sda3 not /dev/sdb3
Using duplicate PV /dev/sda3 without holders, ignoring /dev/sdb3
Found duplicate PV hWoIFGkvc0iVrbZnzhhqxud6QTeotfmQ: using /dev/sda3 not /dev/sdb3
Using duplicate PV /dev/sda3 without holders, ignoring /dev/sdb3
PV Name /dev/sda3
VG Name mauricfo4-vg
With the patch applied, the partition nodes are removed, LVM only scans
the multipath devices, and the boot finishes successfully:
...
(initramfs) exit
...
(initramfs) ls -l /dev/sd*
brw------- 1 8, 16 /dev/sdb
brw------- 1 8, 0 /dev/sda
initramfs) dmsetup table
No devices found
(initramfs) exit
...
(initramfs) dmsetup table | sort
mauricfo4--vg-root: 0 63750144 linear 252:3 2048
mauricfo4--vg-swap_1: 0 2834432 linear 252:3 63752192
mpatha-part1: 0 14336 linear 252:0 2048
mpatha-part2: 0 499712 linear 252:0 16384
mpatha-part3: 0 66590720 linear 252:0 516096
mpatha: 0 67108864 multipath 0 0 2 1 round-robin 0 1 1 8:0 1 round-robin 0 1 1 8:16 1
(initramfs) lvm pvdisplay | grep Name
...
PV Name /dev/mapper/mpatha-part3
VG Name mauricfo4-vg
(initramfs) exit
...
# lsb_release -c
Codename: xenial
# mount | grep ' / '
/dev/mapper/mauricfo4--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
# lvm pvdisplay | grep Name
PV Name /dev/mapper/mpatha-part3
VG Name mauricfo4-vg
# dmsetup ls --tree -o blkdevname
mauricfo4--vg-swap_1 <dm-5> (252:5)
└─mpatha-part3 <dm-3> (252:3)
└─mpatha <dm-0> (252:0)
├─ <sdb> (8:16)
└─ <sda> (8:0)
mpatha-part2 <dm-2> (252:2)
└─mpatha <dm-0> (252:0)
├─ <sdb> (8:16)
└─ <sda> (8:0)
mpatha-part1 <dm-1> (252:1)
└─mpatha <dm-0> (252:0)
├─ <sdb> (8:16)
└─ <sda> (8:0)
mauricfo4--vg-root <dm-4> (252:4)
└─mpatha-part3 <dm-3> (252:3)
└─mpatha <dm-0> (252:0)
├─ <sdb> (8:16)
└─ <sda> (8:0)
[1] qemu command line: commandline: qemu-system-ppc64 -enable-kvm
-nographic -nodefaults -monitor stdio -serial pty -M pseries -smp
1,cores=1,threads=1 -m 8G -drive file=disk.qcow2,serial=xenialroot
-drive file=disk.qcow2,serial=xenialroot -S
** Patch added: "Patch for Xenial"
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540401/+attachment/4569667/+files/multipath-tools_dm-mpath-lvm_xenial.debdiff
--
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1540401
Title:
ISST-LTE: Ubuntu14.04.4 lpar fails to boot after installation: "The
disk drive for /boot is not ready yet or not present"
Status in multipath-tools package in Ubuntu:
In Progress
Bug description:
== Comment: #0 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21 05:15:30 ==
After installing Ubuntu14.04.4 on a PowerVM lpar with "Use entire disk and setup LVM" option at Partition method installation menu, lpar fails to boot up.
Boot error:
-----------------
The disk drive for /boot is not ready yet or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery
Boot log:
------------
Elapsed time since release of system processors: 42348 mins 4 secs
error: no suitable video mode found.
OF stdout device is: /vdevice/vty at 30000000
Preparing to boot Linux version 4.2.0-25-generic (buildd at bos01-ppc64el-023) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #30~14.04.1-Ubuntu SMP Mon Jan 18 16:25:16 UTC 2016 (Ubuntu 4.2.0-25.30~14.04.1-generic 4.2.6)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 256 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/vmlinux-4.2.0-25-generic root=/dev/mapper/biglp1--vg-root ro splash quiet vt.handoff=7
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 000000000ba90000
alloc_top : 0000000010000000
alloc_top_hi : 0000000010000000
rmo_top : 0000000010000000
ram_top : 0000000010000000
instantiating rtas at 0x000000000ec10000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x000000000baa0000 -> 0x000000000baa16b5
Device tree struct 0x000000000bab0000 -> 0x000000000bae0000
Quiescing Open Firmware ...
Booting Linux via __start() ...
-> smp_release_cpus()
spinning_secondaries = 31
<- smp_release_cpus()
<- setup_system()
[ 9.209318] device-mapper: table: 252:2: multipath: error getting device
[ 9.235875] device-mapper: table: 252:2: multipath: error getting device
[ 9.350003] device-mapper: table: 252:6: multipath: error getting device
[ 9.440078] device-mapper: table: 252:6: multipath: error getting device
[ 9.499595] device-mapper: table: 252:6: multipath: error getting device
[ 9.570007] device-mapper: table: 252:6: multipath: error getting device
[ 9.689502] device-mapper: table: 252:6: multipath: error getting device
[ 9.769905] device-mapper: table: 252:6: multipath: error getting device
[ 9.829579] device-mapper: table: 252:6: multipath: error getting device
[ 9.869910] device-mapper: table: 252:6: multipath: error getting device
[ 9.929757] device-mapper: table: 252:6: multipath: error getting device
[ 9.971988] device-mapper: table: 252:6: multipath: error getting device
* Stopping Send an event to indicate plymouth is up [ OK ]
* Starting Mount filesystems on boot [ OK ]
* Starting Populate /dev filesystem [ OK ]
* Starting Populate and link to /run filesystem [ OK ]
* Stopping Populate /dev filesystem [ OK ]
* Stopping Populate and link to /run filesystem [ OK ]
* Stopping Track if upstart is running in a container [ OK ]
* Starting Signal sysvinit that the rootfs is mounted [ OK ]
* Starting Initialize or finalize resolvconf [ OK ]
* Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
* Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
* Starting Bridge udev events into upstart [ OK ]
* Starting Signal sysvinit that remote filesystems are mounted [ OK ]
* Starting device node and kernel event manager [ OK ]
* Starting Clean /tmp directory [ OK ]
* Stopping Clean /tmp directory [ OK ]
* Starting load modules from /etc/modules [ OK ]
* Starting cold plug devices [ OK ]
* Starting log initial device creation [ OK ]
* Stopping load modules from /etc/modules [ OK ]
* Starting Uncomplicated firewall [ OK ]
The disk drive for /boot is not ready yet or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery -----> BOOT Stops here...
Manually aborting the boot (by pressing Shift+M) for manual recovery, provides the lpar prompt.
uname -a:
---------------
root at biglp1:~# uname -a
Linux biglp1 4.2.0-25-generic #30~14.04.1-Ubuntu SMP Mon Jan 18 16:25:16 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
Contents of /etc/fstab:
----------------------
root at biglp1:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/mapper/biglp1--vg-root / ext4 errors=remount-ro 0 1
/dev/mapper/mpath0-part2 /boot ext2 defaults 0 2
#/dev/mapper/biglp1--vg-swap_1 none swap sw 0 0
/dev/mapper/cryptswap1 none swap sw 0 0
df output:
-------------
root at biglp1:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 3046528 512 3046016 1% /dev
tmpfs 619328 5696 613632 1% /run
/dev/mapper/biglp1--vg-root 24308868 1087836 21963152 5% /
none 64 0 64 0% /sys/fs/cgroup
none 5120 0 5120 0% /run/lock
none 3096576 0 3096576 0% /run/shm
none 102400 0 102400 0% /run/user
Steps to recreate:
------------------------
1. Install Ubuntu14.04.4 on a PowerVM lpar having multipath disks.
2. While installing choose "Use entire disk and setup LVM" at disk partitioning installation menu.
3. After installation, lpar fails to mount /boot partition.
Contact info:
----------------
Manju (manjuhr1 at in.ibm.com) A.P (apundt at us.ibm.com)
== Comment: #2 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21
05:24:03 ==
== Comment: #3 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21 05:24:45 ==
== Comment: #4 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21 05:25:40 ==
== Comment: #5 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21 05:28:15 ==
== Comment: #6 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21 05:29:24 ==
== Comment: #8 - Manjunatha H R <manjuhr1 at in.ibm.com> - 2016-01-21 05:37:27 ==
root at biglp1:~# dpkg -l |grep multipath
ii multipath-tools 0.4.9-3ubuntu7.7 ppc64el maintain multipath block device access
ii multipath-tools-boot 0.4.9-3ubuntu7.7 all Support booting from multipath devices
root at biglp1:~# dpkg -l|grep lvm
ii lvm2 2.02.98-6ubuntu2 ppc64el Linux Logical Volume Manager
== Comment: #19 - Mauricio Faria De Oliveira <mauricfo at br.ibm.com> - 2016-01-25 14:55:23 ==
The problem matches the suspicion: the LVM detection is happening before multipathd grabs the individual paths, and then the creation of the multipath map /dev/mapper/mpath0 fails, then /boot fails to mount as it's specified as /dev/mapper/mpath0-part2 in /etc/fstab:
root at biglp1:~# pvdisplay | grep Name
Found duplicate PV xkHFzaklbXIhfOQfI74LdjE2yPErlQtc: using /dev/sdu3 not /dev/sda3
Found duplicate PV xkHFzaklbXIhfOQfI74LdjE2yPErlQtc: using /dev/sdf3 not /dev/sdu3
Found duplicate PV xkHFzaklbXIhfOQfI74LdjE2yPErlQtc: using /dev/sdz3 not /dev/sdf3
Found duplicate PV xkHFzaklbXIhfOQfI74LdjE2yPErlQtc: using /dev/sdk3 not /dev/sdz3
Found duplicate PV xkHFzaklbXIhfOQfI74LdjE2yPErlQtc: using /dev/sdp3 not /dev/sdk3
PV Name /dev/sdp3
VG Name biglp1-vg
root at biglp1:~# multipath -v3 /dev/sdp
...
Jan 25 13:38:06 | 36005076308ffc54b000000000000003f: alias_prefix = mpath (internal default)
Jan 25 13:38:06 | Found matching wwid [36005076308ffc54b000000000000003f] in bindings file. Setting alias to mpath0
Jan 25 13:38:06 | sdp: ownership set to mpath0
...
Jan 25 13:38:06 | sda: ownership set to mpath0
...
Jan 25 13:38:06 | sdf: ownership set to mpath0
...
Jan 25 13:38:06 | sdk: ownership set to mpath0
...
Jan 25 13:38:06 | sdu: ownership set to mpath0
...
Jan 25 13:38:06 | sdz: ownership set to mpath0
...
Jan 25 13:38:06 | mpath0: pgfailover = -1 (internal default)
Jan 25 13:38:06 | mpath0: pgpolicy = multibus (controller setting)
Jan 25 13:38:06 | mpath0: selector = round-robin 0 (controller setting)
Jan 25 13:38:06 | mpath0: features = 1 queue_if_no_path (controller setting)
Jan 25 13:38:06 | mpath0: hwhandler = 0 (controller setting)
Jan 25 13:38:06 | mpath0: rr_weight = 1 (controller setting)
Jan 25 13:38:06 | mpath0: minio = 1000 (controller setting)
Jan 25 13:38:06 | mpath0: no_path_retry = NONE (internal default)
Jan 25 13:38:06 | pg_timeout = NONE (internal default)
Jan 25 13:38:06 | mpath0: set ACT_CREATE (map does not exist)
[ 213.998298] device-mapper: table: 252:6: multipath: error getting device
[ 214.030777] device-mapper: table: 252:6: multipath: error getting device
Jan 25 13:38:06 | mpath0: domap (0) failure for create/reload map
Looking into a patch for this.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1540401/+subscriptions
More information about the Ubuntu-sponsors
mailing list