[Bug 1374999] Re: iSCSI volume detach does not correctly remove the multipath device descriptors
Rafael David Tinoco
rafael.tinoco at canonical.com
Fri Nov 14 14:16:42 UTC 2014
I have created the following PPA:
https://launchpad.net/~inaddy/+archive/ubuntu/lp1374999
containing a hotfixed version of nova for Trusty.
I'll keep this PPA available, updating when needed, until we get
upstream fix accepted and backported. As soon as the fix is backported
we can ask for a "SRU" for Trusty's nova version.
I'm assigning this case temporary to Felipe (responsible to propose the
fix upstream) while I'm on vacation.
Let me know if there any issues.
Thank you.
** Changed in: nova
Assignee: Rafael David Tinoco (inaddy) => (unassigned)
--
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1374999
Title:
iSCSI volume detach does not correctly remove the multipath device
descriptors
Status in OpenStack Compute (Nova):
Confirmed
Status in “nova” package in Ubuntu:
Confirmed
Status in “nova” source package in Trusty:
Confirmed
Bug description:
iSCSI volume detach does not correctly remove the multipath device
descriptors
tested environment:
nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.
I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths.
Here is the terminal log for last volume detach.
openstack at W1DEV103:~/devstack$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
|
ID
| Status | Name | Size | Volume Type | Bootable |
Attached to
|
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
None
| false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
openstack at W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:38:13 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- 4:0:0:42 sdb 8:16 active undef running
| |- 5:0:0:42 sdd 8:48 active undef running
| |- 6:0:0:42 sdf 8:80 active undef running
| `- 7:0:0:42 sdh 8:112 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- 11:0:0:42 sdp 8:240 active undef running
|- 8:0:0:42 sdj 8:144 active undef running
|- 9:0:0:42 sdl 8:176 active undef running
`- 10:0:0:42 sdn 8:208 active undef running
openstack at W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:38:19 JST 2014
tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6
openstack at W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
56a63288-5cc0-4f5c-9197-cde731172dd8
openstack at W1DEV103:~/devstack$
openstack at W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
|
ID
| Status | Name | Size | Volume Type | Bootable |
Attached to
|
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
None
| false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
openstack at W1DEV103:~/devstack$
openstack at W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
ID
| Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
None
| false |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
openstack at W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:39:23 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 ,
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| `- #:#:#:# - #:# active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
`- #:#:#:# - #:# active undef running
openstack at W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:39:27 JST 2014
iscsiadm: No active sessions.
Then I manually removed the multipaths,
openstack at W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
openstack at W1CN103:/etc/iscsi$ sudo multipath -l
openstack at W1CN103:/etc/iscsi$
I think the problem is in,
virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):
End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
self._remove_multipath_device_descriptor(multipath_device)
return
Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_multipath_device_descriptor due to following if statement.
if not in_use:
# disconnect if no other multipath devices with same iqn
self._disconnect_mpath(iscsi_properties, ips_iqns)
return
It just disconnect them but not remove them.
One of the reasons why we have to remove them is,
https://bugs.launchpad.net/nova/+bug/1223975
IMO, we should call _remove_multipath_device_descriptor in above if
statement before return.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1374999/+subscriptions
More information about the Ubuntu-sponsors
mailing list