[Bug 1475411] Re: During post_live_migration the nova libvirt driver assumes that the destination connection info is the same as the source, which is not always true

Billy Olsen billy.olsen at canonical.com
Fri Jul 1 23:38:40 UTC 2016


** Description changed:

+ [Impact]
+ 
  The post_live_migration step for Nova libvirt driver is currently making
  a bad assumption about the source and destination connector information.
  The destination connection info may be different from the source which
  ends up causing LUNs to be left dangling on the source as the BDM has
  overridden the connection info with that of the destination.
  
  Code section where this problem is occuring:
  
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6036
  
  At line 6038 the potentially wrong connection info will be passed to
  _disconnect_volume which then ends up not finding the proper LUNs to
  remove (and potentially removes the LUNs for a different volume
  instead).
  
  By adding debug logging after line 6036 and then comparing that to the
  connection info of the source host (by making a call to Cinder's
  initialize_connection API) you can see that the connection info does not
  match:
  
  http://paste.openstack.org/show/TjBHyPhidRuLlrxuGktz/
  
  Version of nova being used:
  
  commit 35375133398d862a61334783c1e7a90b95f34cdb
  Merge: 83623dd b2c5542
  Author: Jenkins <jenkins at review.openstack.org>
  Date:   Thu Jul 16 02:01:05 2015 +0000
  
      Merge "Port crypto to Python 3"
+ 
+ 
+ [Test Case]
+ 
+ Live migrate an instance which is connected to a volume through multi-
+ path in which the source and target connection information is not the
+ same. Verify that the correct device/LUN is removed (instead of wrong
+ one).
+ 
+ [Regression Potential]
+ 
+ The regression potential is small as it has run in newer versions of
+ nova for awhile now (since Juno, the release immediately following
+ Icehouse). If a regression were to occur it would likely prevent a live
+ migration from completing (failing in the post processing), leaving the
+ instance in an error state. However, it should be migrated to the target
+ hypervisor with access to the LUN so it would require manual cleanup of
+ the lun at the source hypervisor and a reset of the instance state to
+ active.

** Tags removed: verification-needed
** Tags added: verification-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1475411

Title:
  During post_live_migration the nova libvirt driver assumes that the
  destination connection info is the same as the source, which is not
  always true

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) juno series:
  Fix Released
Status in OpenStack Compute (nova) kilo series:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Trusty:
  Fix Committed

Bug description:
  [Impact]

  The post_live_migration step for Nova libvirt driver is currently
  making a bad assumption about the source and destination connector
  information. The destination connection info may be different from the
  source which ends up causing LUNs to be left dangling on the source as
  the BDM has overridden the connection info with that of the
  destination.

  Code section where this problem is occuring:

  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6036

  At line 6038 the potentially wrong connection info will be passed to
  _disconnect_volume which then ends up not finding the proper LUNs to
  remove (and potentially removes the LUNs for a different volume
  instead).

  By adding debug logging after line 6036 and then comparing that to the
  connection info of the source host (by making a call to Cinder's
  initialize_connection API) you can see that the connection info does
  not match:

  http://paste.openstack.org/show/TjBHyPhidRuLlrxuGktz/

  Version of nova being used:

  commit 35375133398d862a61334783c1e7a90b95f34cdb
  Merge: 83623dd b2c5542
  Author: Jenkins <jenkins at review.openstack.org>
  Date:   Thu Jul 16 02:01:05 2015 +0000

      Merge "Port crypto to Python 3"

  
  [Test Case]

  Live migrate an instance which is connected to a volume through multi-
  path in which the source and target connection information is not the
  same. Verify that the correct device/LUN is removed (instead of wrong
  one).

  [Regression Potential]

  The regression potential is small as it has run in newer versions of
  nova for awhile now (since Juno, the release immediately following
  Icehouse). If a regression were to occur it would likely prevent a
  live migration from completing (failing in the post processing),
  leaving the instance in an error state. However, it should be migrated
  to the target hypervisor with access to the LUN so it would require
  manual cleanup of the lun at the source hypervisor and a reset of the
  instance state to active.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1475411/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list