[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
Alasdair G. Kergon
agk at redhat.com
Mon Jul 9 17:11:14 UTC 2012
On Mon, Jul 09, 2012 at 04:33:25PM -0000, Peter Petrakis wrote:
> I'm not familiar with the cookie feature you mentioned, but we appear to have some of the
> capability here. I haven't compared it to upstream yet.
The code issuing the removal ioctl specfies a cookie (a semaphore with a
positive value). It then waits until the semaphore drops to 0.
The last thing the udev rules do after they've updated /dev is decrement
the semaphore, which wakes up the original code.
There are libdevmapper calls and dmsetup commands to update/query cookie
state.
udev 'watch' rules can run out-of-band and need careful configuration to
avoid problems.
If libvirt is issuing dm ioctls or dmsetup commands itself then it might
also need to be udevcookie-aware - not something that's arisen in any
discussions yet - there is some default udevcookie handling built into
dmsetup commands.
I was assured the upstream multipath udevcookie support was complete,
but I've not verified that personally - perhaps we have a situation
that's overlooked here.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/585027
Title:
Race condition with dmsetup causes 'map already present' messages
Status in “multipath-tools” package in Ubuntu:
Invalid
Bug description:
Binary package hint: multipath-tools
Environment:
njones at fantail:~$ lsb_release -rd
Description: Ubuntu 10.04 LTS
Release: 10.04
njones at fantail:~$ apt-cache policy multipath-tools
multipath-tools:
Installed: 0.4.8-14ubuntu4
Candidate: 0.4.8-14ubuntu4
Version table:
*** 0.4.8-14ubuntu4 0
500 http://nz.archive.ubuntu.com/ubuntu/ lucid/main Packages
100 /var/lib/dpkg/status
Issue:
While reproducing
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/571093 both
myself and Dustin Kirkland encountered an apparent race condition in
multipath and dmsetup.
We were attempting variants of 'while true; do sudo multipath -F; sudo
multipath -v4 ; done' to create a set of udev add/remove events and
noticed that shortly after starting, no output appeared within the
'udevadm monitor' command.
We observed that the output from a: multipath -v4 was similar to:
root at fantail:~# multipath -v4
Discover device /sys/block/ram0
ram0: device node name blacklisted
Discover device /sys/block/ram1
ram1: device node name blacklisted
Discover device /sys/block/ram2
ram2: device node name blacklisted
Discover device /sys/block/ram3
ram3: device node name blacklisted
Discover device /sys/block/ram4
ram4: device node name blacklisted
Discover device /sys/block/ram5
ram5: device node name blacklisted
Discover device /sys/block/ram6
ram6: device node name blacklisted
Discover device /sys/block/ram7
ram7: device node name blacklisted
Discover device /sys/block/ram8
ram8: device node name blacklisted
Discover device /sys/block/ram9
ram9: device node name blacklisted
Discover device /sys/block/ram10
ram10: device node name blacklisted
Discover device /sys/block/ram11
ram11: device node name blacklisted
Discover device /sys/block/ram12
ram12: device node name blacklisted
Discover device /sys/block/ram13
ram13: device node name blacklisted
Discover device /sys/block/ram14
ram14: device node name blacklisted
Discover device /sys/block/ram15
ram15: device node name blacklisted
Discover device /sys/block/loop0
loop0: device node name blacklisted
Discover device /sys/block/loop1
loop1: device node name blacklisted
Discover device /sys/block/loop2
loop2: device node name blacklisted
Discover device /sys/block/loop3
loop3: device node name blacklisted
Discover device /sys/block/loop4
loop4: device node name blacklisted
Discover device /sys/block/loop5
loop5: device node name blacklisted
Discover device /sys/block/loop6
loop6: device node name blacklisted
Discover device /sys/block/loop7
loop7: device node name blacklisted
Discover device /sys/block/sda
sda: not found in pathvec
sda: mask = 0x1f
sda: dev_t = 8:0
sda: size = 625142448
sda: subsystem = scsi
sda: vendor = ATA
sda: product = WDC WD3200AAKS-0
sda: rev = 12.0
sda: h:b:t:l = 0:0:0:0
sda: serial = WD-WCAS70000070
sda: getprio = NULL (internal default)
sda: prio = 1
sda: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default)
sda: uid = 1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070 (callout)
Discover device /sys/block/sr0
sr0: device node name blacklisted
Discover device /sys/block/sdb
sdb: not found in pathvec
sdb: mask = 0x1f
sdb: dev_t = 8:16
sdb: size = 15679488
sdb: subsystem = scsi
sdb: vendor = Kingston
sdb: product = DataTraveler G2
sdb: rev = PMAP
sdb: h:b:t:l = 6:0:0:0
sdb: serial =
sdb: getprio = NULL (internal default)
sdb: prio = 1
sdb: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default)
error calling out /lib/udev/scsi_id -g -u -d /dev/sdb
Discover device /sys/block/sdc
sdc: not found in pathvec
sdc: mask = 0x1f
sdc: dev_t = 8:32
sdc: size = 41943040
sdc: subsystem = scsi
sdc: vendor = IET
sdc: product = VIRTUAL-DISK
sdc: rev = 0
sdc: h:b:t:l = 7:0:0:0
sdc: serial =
sdc: getprio = NULL (internal default)
sdc: prio = 1
sdc: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default)
error calling out /lib/udev/scsi_id -g -u -d /dev/sdc
Discover device /sys/block/dm-0
dm-0: device node name blacklisted
Discovery status 0
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070 0:0:0:0 sda 8:0 1 [undef][undef
6:0:0:0 sdb 8:16 1 [undef][undef
7:0:0:0 sdc 8:32 1 [undef][undef
sda: ownership set to 1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070
sda: not found in pathvec
sda: mask = 0xc
sda: path checker = directio (config file default)
directio: called for 800
directio: called in synchronous mode
directio: starting new request
directio: async io getevents returns 1 (errno=No such file or directory)
directio: io finished 4096/0
sda: state = 2
sda: prio = 1
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: pgfailover = -1 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: pgpolicy = failover (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: selector = round-robin 0 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: features = 0 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: hwhandler = 0 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: rr_weight = 1 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: minio = 1000 (config file default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: no_path_retry = NONE (internal default)
pg_timeout = NONE (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: set ACT_CREATE (map does not exist)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: map already present
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: domap (0) failure for create/reload map
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: remove multipath map
sda: orphaned
And a device was left behind in /dev/mapper:
njones at fantail:~$ ll /dev/mapper/
total 0
drwxr-xr-x 2 root root 80 2010-05-25 04:25 ./
drwxr-xr-x 18 root root 4080 2010-05-25 04:25 ../
brw-rw---- 1 root disk 252, 0 2010-05-25 04:25 1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070
crw-rw---- 1 root root 10, 59 2010-05-22 10:35 control
$ sudo dmsetup ls
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070 (252, 0)
We had to perform 'dmsetup remove <id>' to clear the 'map already
present' message.
While I admit that this is a very specific case, it is still a race
condition and does warrant a bug report.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions
More information about the foundations-bugs
mailing list