[Bug 1900775] Re: Cinder fails to create image-based volume if mirroring is enabled

Wed Feb 24 19:18:25 UTC 2021

verified successfully

== groovy-proposed ==
root at g1:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume

/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
.
----------------------------------------------------------------------
Ran 1 test in 1.403s

OK
root at g1:/usr/lib/python3/dist-packages# 
root at g1:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:17.0.1-0ubuntu2
  Candidate: 2:17.0.1-0ubuntu2
  Version table:
 *** 2:17.0.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 Packages

== focal-proposed ==
root at f1:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
.
----------------------------------------------------------------------
Ran 1 test in 0.901s

OK
root at f1:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:16.2.1-0ubuntu2
  Candidate: 2:16.2.1-0ubuntu2
  Version table:
 *** 2:16.2.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages

== ussuri-proposed ==
root at bu:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
.
----------------------------------------------------------------------
Ran 1 test in 1.201s

OK
root at bu:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:16.2.1-0ubuntu2~cloud0
  Candidate: 2:16.2.1-0ubuntu2~cloud0
  Version table:
 *** 2:16.2.1-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages

== train-proposed ==
root at bt:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
/usr/lib/python3/dist-packages/migrate/versioning/script/py.py:144: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  if not inspect.getargspec(script_func)[0]:
.
----------------------------------------------------------------------
Ran 1 test in 0.733s

OK
root at bt:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:15.4.1-0ubuntu1~cloud1
  Candidate: 2:15.4.1-0ubuntu1~cloud1
  Version table:
 *** 2:15.4.1-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages

== stein-proposed ==
root at bs:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
/usr/lib/python3/dist-packages/migrate/versioning/script/py.py:144: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  if not inspect.getargspec(script_func)[0]:
.
----------------------------------------------------------------------
Ran 1 test in 1.546s

OK
root at bs:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:14.3.1-0ubuntu1~cloud1
  Candidate: 2:14.3.1-0ubuntu1~cloud1
  Version table:
 *** 2:14.3.1-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64 Packages

== bionic-proposed ==
root at b1:/usr/lib/python2.7/dist-packages# python -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
.
----------------------------------------------------------------------
Ran 1 test in 1.304s

OK
root at b1:/usr/lib/python2.7/dist-packages# apt policy python-cinder
python-cinder:
  Installed: 2:12.0.10-0ubuntu2
  Candidate: 2:12.0.10-0ubuntu2
  Version table:
 *** 2:12.0.10-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages

== queens-proposed ==
root at x1:/usr/lib/python2.7# python -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
.
----------------------------------------------------------------------
Ran 1 test in 1.144s

OK
root at x1:/usr/lib/python2.7# apt policy python-cinder
python-cinder:
  Installed: 2:12.0.10-0ubuntu2~cloud0
  Candidate: 2:12.0.10-0ubuntu2~cloud0
  Version table:
 *** 2:12.0.10-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-proposed/queens/main amd64 Packages

** Tags removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-queens-needed verification-stein-needed verification-train-needed verification-ussuri-needed verification-victoria-needed
** Tags added: verification-done verification-done-bionic verification-done-focal verification-done-groovy verification-queens-done verification-stein-done verification-train-done verification-ussuri-done verification-victoria-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1900775

Title:
  Cinder fails to create image-based volume if mirroring is enabled

Status in charm-ceph-rbd-mirror:
  Invalid
Status in Cinder:
  In Progress
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  Fix Committed
Status in Ubuntu Cloud Archive ussuri series:
  Fix Committed
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in cinder package in Ubuntu:
  Fix Released
Status in cinder source package in Xenial:
  Triaged
Status in cinder source package in Bionic:
  Fix Committed
Status in cinder source package in Focal:
  Fix Committed
Status in cinder source package in Groovy:
  Fix Committed

Bug description:
  [Impact]

  OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way.
  Cinder has Ceph as backend for volumes.

  Creating a volume from qcow2 image.
  Current flow is the following:
  1. Cinder creates empty volume in Ceph
  2. Cinder downloads the image
  3. Image is converted to raw
  4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611
  5. Cinder performs "rbd import" using unpacked raw image as source.

  Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second.
  It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error.
  The only way to fix this behaviour - disable mirroring of the backend pool which is not desired.

  [Test Case]
  This is a light-weight test to ensure the code is working as expected, using the unit test from the patch:

  lxc launch ubuntu-daily:hirsute h1
  lxc exec h1 /bin/bash
  root at h1:~# sudo apt install python3-cinder
  root at h1:~# cd /usr/lib/python3/dist-packages/
  root at h1:~# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
  /usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
    self._legacy_facade = LegacyEngineFacade(None, _factory=self)
  .
  ----------------------------------------------------------------------
  Ran 1 test in 0.701s

  OK

  The test will fail if the fixed code is not installed.

  [Regression Potential]
  This is a very minimal change that simply adds a retry when exception.VolumeIsBusy is encountered. The frequency and number of retries are configurable via rados_connection_interval and rados_connection_retries. Worst case scenario, if these are exceeded, the original error will be encountered.

  [Discussion]
  This is accompanied by a unit test fix for: https://pad.lv/1913607

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-rbd-mirror/+bug/1900775/+subscriptions