[Bug 1178721] Re: multipathd fails to create mappings when multipath.conf is present
Jernej Jakob
jernej.jakob at gmail.com
Mon Jun 10 19:33:24 UTC 2013
Thank you for your help. I wasn't aware of the limitation
(incompatibility) caused by co-deploying LVM and multipath, it was my
assumption that multipath would be configured to take precedence over
LVM for having PVs on multipath block devices, as having it the other
way around, as in multiple paths over LVM logical volumes would not make
sense. I can assume that this would be a pretty common scenario, as for
example, you may have one volume mapped from SAN over FC on which the
tools that LVM offers would be valuable.
It would seem that a simple check at boot time, checking whether LVM and
multipath are co-deployed, starting multipath first, letting it create
paths and then adding those found to LVM's filter.
I understand however that this is in no way common usage and requires a
skillful enough administrator to be aware of the possible scenarios
caused.
Regarding the multipath config itself, this is the default config, functionally the same as the one proposed for MSA2012fc by HP.
The virtual one is of course bogus... not to be used, just to demonstrate the failure mode. The actual servers will have a dual-port FC HBA with each port going to separate zones on a switch, with itself being connected with 2 cables each to the 2 controllers (4 in total) of the SAN, so there will be 2 paths for each volume. This is required for this SAN in order to have redundancy for controller failure.
I just tried this with the physical server, and it still doesn't work as
it should. In lvm.conf, blacklisted all the sd* devices and allowed only
/dev/disk/by-id. Or should even the id's of the MP volumes be
blacklisted, only allowing the one single PV's id?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1178721
Title:
multipathd fails to create mappings when multipath.conf is present
Status in “multipath-tools” package in Ubuntu:
Triaged
Bug description:
I have a SAN connected via a dual-port HBA that I'm trying to get to
work with multipath.
I'm running 12.04.2 installed via netboot (clean install).
Setup options: partitioning=guided with LVM (onto internal DRAC 6/i RAID) packages=Basic server,OpenSSH,Virtual machine host
Out of the box things work okay:
root at PE2950:~# multipath -ll
3600c0ff000d5ae56aabb855101000000 dm-3 HP,MSA2012fc
size=1.4T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:1 sdb 8:16 active ready running
`- 4:0:1:1 sdc 8:32 active ready running
mappings are created:
root at PE2950:~# ll /dev/mapper/
total 0
drwxr-xr-x 2 root root 140 maj 10 16:57 ./
drwxr-xr-x 16 root root 4460 maj 10 16:57 ../
brw-rw---- 1 root disk 252, 2 maj 10 16:57 360022190ba84f800113787c003d70474
lrwxrwxrwx 1 root root 7 maj 10 16:57 3600c0ff000d5ae56aabb855101000000 -> ../dm-3
crw------- 1 root root 10, 236 maj 9 23:13 control
lrwxrwxrwx 1 root root 7 maj 9 23:14 PE2950-root -> ../dm-1
lrwxrwxrwx 1 root root 7 maj 9 23:14 PE2950-swap -> ../dm-0
Then I created /etc/multipath.conf with following contents (mostly
copied from running multipathd config):
multipath.conf:
------------------
defaults {
verbosity 2
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^dcssblk[0-9]*"
devnode "^cciss!c[0-9]d[0-9]*"
device {
vendor "DGC"
product "LUNZ"
}
device {
vendor "EMC"
product "LUNZ"
}
device {
vendor "IBM"
product "S/390.*"
}
device {
vendor "IBM"
product "S/390.*"
}
device {
vendor "STK"
product "Universal Xport"
}
device {
vendor "DELL"
product "PERC.*"
}
}
blacklist_exceptions {
}
devices {
device {
vendor "HP"
product "MSA2[02]12fc|MSA2012i"
path_grouping_policy multibus
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_selector round-robin 0
path_checker tur
checker tur
features "0"
hardware_handler "0"
prio const
failback immediate
no_path_retry 18
rr_min_io 100
}
}
multipaths {
}
(point is in trying to turn off the feature 'queue_if_no_path')
root at PE2950:~# ll /etc/multipath.conf
-rw-r--r-- 1 root root 0 maj 10 17:14 /etc/multipath.conf
Have to reboot:
root at PE2950:~$ reboot
Things get ugly - no more mappings:
root at PE2950:~# multipath -ll
root at PE2950:~#
root at PE2950:~# ls -la /dev/mapper/
total 0
drwxr-xr-x 2 root root 100 maj 10 17:23 .
drwxr-xr-x 16 root root 4420 maj 10 17:23 ..
crw------- 1 root root 10, 236 maj 10 17:22 control
lrwxrwxrwx 1 root root 7 maj 10 17:22 PE2950-root -> ../dm-1
lrwxrwxrwx 1 root root 7 maj 10 17:22 PE2950-swap -> ../dm-0
syslog:
--snip--
May 10 17:23:33 PE2950 udevd[380]: '/lib/udev/dmsetup_env 252 2'(err) 'Device does not exist.'
May 10 17:23:33 PE2950 udevd[380]: '/lib/udev/dmsetup_env 252 2'(err) 'Command failed'
May 10 17:23:33 PE2950 scsi_id[1156]: custom logging function 0x7f93cbe8d010 registered
May 10 17:23:33 PE2950 udevd[380]: '/lib/udev/dmsetup_env 252 2'(err) 'Device does not exist.'
May 10 17:23:33 PE2950 udevd[380]: '/lib/udev/dmsetup_env 252 2'(err) 'Command failed'
May 10 17:23:33 PE2950 udevd[380]: '/lib/udev/dmsetup_env 252 2'(out) 'DM_NAME='
May 10 17:23:33 PE2950 udevd[380]: '/lib/udev/dmsetup_env 252 2' [1154] exit with return code 0
--snip--
May 10 17:23:33 PE2950 kernel: [ 90.706237] device-mapper: multipath round-robin: version 1.0.0 loaded
May 10 17:23:33 PE2950 kernel: [ 90.706403] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.706405] device-mapper: ioctl: error adding target to table
--snip--
May 10 17:23:33 PE2950 udevd[379]: no db file to read /run/udev/data/+bdi:252:2: No such file or directory
May 10 17:23:33 PE2950 udevd[378]: '/sbin/blkid -o udev -p /dev/.tmp-block-252:2'(err) 'error: /dev/.tmp-block-252:2: No such device or address'
--snip--
May 10 17:23:33 PE2950 udevd[378]: '/lib/udev/dmsetup_env 252 2'(err) 'Device does not exist.'
May 10 17:23:33 PE2950 udevd[378]: '/lib/udev/dmsetup_env 252 2'(err) 'Command failed'
May 10 17:23:33 PE2950 udevd[378]: '/lib/udev/dmsetup_env 252 2'(err) 'Device does not exist.'
May 10 17:23:33 PE2950 udevd[378]: '/lib/udev/dmsetup_env 252 2'(err) 'Command failed'
May 10 17:23:33 PE2950 udevd[378]: '/lib/udev/dmsetup_env 252 2'(out) 'DM_NAME='
May 10 17:23:33 PE2950 udevd[378]: '/lib/udev/dmsetup_env 252 2' [1173] exit with return code 0
--snip--
May 10 17:23:33 PE2950 udevd[1183]: starting '/sbin/dmsetup info -j 252 -m 2 -c --nameprefixes --noheadings --rows -o suspended'
May 10 17:23:33 PE2950 udevd[378]: '/sbin/dmsetup info -j 252 -m 2 -c --nameprefixes --noheadings --rows -o suspended'(err) 'Device does not exist.'
May 10 17:23:33 PE2950 udevd[378]: '/sbin/dmsetup info -j 252 -m 2 -c --nameprefixes --noheadings --rows -o suspended'(err) 'Command failed'
May 10 17:23:33 PE2950 udevd[378]: '/sbin/dmsetup info -j 252 -m 2 -c --nameprefixes --noheadings --rows -o suspended' [1183] exit with return code 1
--snip--
May 10 17:23:33 PE2950 kernel: [ 90.712666] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.712668] device-mapper: ioctl: error adding target to table
May 10 17:23:33 PE2950 kernel: [ 90.713856] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.713858] device-mapper: ioctl: error adding target to table
May 10 17:23:33 PE2950 udevd[379]: passed -1 bytes to socket monitor 0x7f133e032330
May 10 17:23:33 PE2950 kernel: [ 90.714250] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.714252] device-mapper: ioctl: error adding target to table
May 10 17:23:33 PE2950 kernel: [ 90.717242] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.717245] device-mapper: ioctl: error adding target to table
May 10 17:23:33 PE2950 kernel: [ 90.717840] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.717842] device-mapper: ioctl: error adding target to table
May 10 17:23:33 PE2950 kernel: [ 90.747257] device-mapper: table: 252:2: multipath: not enough path parameters
May 10 17:23:33 PE2950 kernel: [ 90.747259] device-mapper: ioctl: error adding target to table
(the whole syslog is attached)
When things were working, I also tried creating a multipath storage pool in virt-manager pointing to /dev/mapper, and now libvirtd is crashing. (should I file a seperate bug report for this? or wait if this turns out to be a related problem)
May 10 17:04:26 PE2950 kernel: [64283.865145] init: libvirt-bin main process (24372) killed by SEGV signal
May 10 17:04:26 PE2950 kernel: [64283.865186] init: libvirt-bin main process ended, respawning
May 10 17:04:26 PE2950 kernel: [64284.030613] init: libvirt-bin main process (24469) killed by SEGV signal
May 10 17:04:26 PE2950 kernel: [64284.030645] init: libvirt-bin main process ended, respawning
May 10 17:04:26 PE2950 kernel: [64284.189976] init: libvirt-bin main process (24515) killed by SEGV signal
May 10 17:04:26 PE2950 kernel: [64284.190002] init: libvirt-bin main process ended, respawning
May 10 17:04:26 PE2950 kernel: [64284.353151] init: libvirt-bin main process (24559) killed by SEGV signal
May 10 17:04:26 PE2950 kernel: [64284.353182] init: libvirt-bin main process ended, respawning
May 10 17:04:27 PE2950 kernel: [64284.520619] init: libvirt-bin main process (24613) killed by SEGV signal
May 10 17:04:27 PE2950 kernel: [64284.520645] init: libvirt-bin main process ended, respawning
May 10 17:04:27 PE2950 kernel: [64284.682288] init: libvirt-bin main process (24657) killed by SEGV signal
May 10 17:04:27 PE2950 kernel: [64284.682321] init: libvirt-bin main process ended, respawning
May 10 17:04:27 PE2950 kernel: [64284.844660] init: libvirt-bin main process (24701) killed by SEGV signal
May 10 17:04:27 PE2950 kernel: [64284.844690] init: libvirt-bin main process ended, respawning
May 10 17:04:27 PE2950 kernel: [64285.006189] init: libvirt-bin main process (24745) killed by SEGV signal
May 10 17:04:27 PE2950 kernel: [64285.006214] init: libvirt-bin main process ended, respawning
May 10 17:04:27 PE2950 kernel: [64285.195574] init: libvirt-bin main process (24789) killed by SEGV signal
May 10 17:04:27 PE2950 kernel: [64285.195605] init: libvirt-bin main process ended, respawning
May 10 17:04:27 PE2950 kernel: [64285.366351] init: libvirt-bin main process (24833) killed by SEGV signal
May 10 17:04:27 PE2950 kernel: [64285.366379] init: libvirt-bin main process ended, respawning
May 10 17:04:28 PE2950 kernel: [64285.528396] init: libvirt-bin main process (24892) killed by SEGV signal
May 10 17:04:28 PE2950 kernel: [64285.528426] init: libvirt-bin respawning too fast, stopped
I reproduced the bug with a virtual machine install without the SAN,
so this is definitively a problem with multipath-tools or udev.
Steps to reproduce:
1.create VM with three disks: virtio for root, two SCSI pointing to same file for multipath
(same file is not really neccesary, as it detects as two seperate disks anyway)
2.install 12.04.2 amd64(from PXE netboot)
select guided partitioning with LVM
install packages Basic Ubuntu server,OpenSSH,Virtual machine host
3.after booting create multipath.conf, then reboot
4.bug appears
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1178721/+subscriptions
More information about the foundations-bugs
mailing list