[Bug 1374999] Re: iSCSI volume detach does not correctly remove the multipath device descriptors

Billy Olsen billy.olsen at canonical.com
Mon Jan 11 22:16:06 UTC 2016


** Also affects: cloud-archive
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1374999

Title:
  iSCSI volume detach does not correctly remove the multipath device
  descriptors

Status in Ubuntu Cloud Archive:
  New
Status in OpenStack Compute (nova):
  In Progress
Status in nova package in Ubuntu:
  Triaged
Status in nova source package in Trusty:
  Triaged

Bug description:
  iSCSI volume detach does not correctly remove the multipath device
  descriptors

  tested environment:
  nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.

   I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully.  3rd volume also successfully detached but ended up with  failed multipaths. 
  Here is the terminal log for last volume detach.

  openstack at W1DEV103:~/devstack$ cinder list
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable |
   Attached to
   |
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
   None
   | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  openstack at W1CN103:/etc/iscsi$ date;sudo multipath -l
  Fri Sep 19 21:38:13 JST 2014
  360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
  size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | |- 4:0:0:42 sdb 8:16 active undef running
  | |- 5:0:0:42 sdd 8:48 active undef running
  | |- 6:0:0:42 sdf 8:80 active undef running
  | `- 7:0:0:42 sdh 8:112 active undef running
  `-+- policy='round-robin 0' prio=-1 status=enabled
  |- 11:0:0:42 sdp 8:240 active undef running
  |- 8:0:0:42 sdj 8:144 active undef running
  |- 9:0:0:42 sdl 8:176 active undef running
  `- 10:0:0:42 sdn 8:208 active undef running
  openstack at W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
  Fri Sep 19 21:38:19 JST 2014
  tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
  tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
  tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
  tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
  tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
  tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
  tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
  tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6

  openstack at W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
  56a63288-5cc0-4f5c-9197-cde731172dd8
  openstack at W1DEV103:~/devstack$
  openstack at W1DEV103:~/devstack$ cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable |
   Attached to
   |
  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
   None
   | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|

  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  openstack at W1DEV103:~/devstack$
  openstack at W1DEV103:~/devstack$ cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
   None
   | false |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |
  openstack at W1CN103:/etc/iscsi$ date;sudo multipath -l
  Fri Sep 19 21:39:23 JST 2014
  360060160cf0036002d1475f6e73fe411 dm-2 ,
  size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | |- #:#:#:# - #:# active undef running
  | |- #:#:#:# - #:# active undef running
  | |- #:#:#:# - #:# active undef running
  | `- #:#:#:# - #:# active undef running
  `-+- policy='round-robin 0' prio=-1 status=enabled
  |- #:#:#:# - #:# active undef running
  |- #:#:#:# - #:# active undef running
  |- #:#:#:# - #:# active undef running
  `- #:#:#:# - #:# active undef running
  openstack at W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
  Fri Sep 19 21:39:27 JST 2014
  iscsiadm: No active sessions.

  Then I manually removed the multipaths,
  openstack at W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
  openstack at W1CN103:/etc/iscsi$ sudo multipath -l
  openstack at W1CN103:/etc/iscsi$

   I think the problem is in,
  virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
   def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):

  End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
          self._remove_multipath_device_descriptor(multipath_device)
          return
  Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one),  this method return without calling _remove_multipath_device_descriptor due to following if statement. 
   
          if not in_use:
              # disconnect if no other multipath devices with same iqn
              self._disconnect_mpath(iscsi_properties, ips_iqns)
              return
  It just disconnect them but not remove them.
  One of the reasons why we have to remove them is,
  https://bugs.launchpad.net/nova/+bug/1223975

  IMO, we should call _remove_multipath_device_descriptor in above if
  statement before return.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1374999/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list