[Bug 1900775] Re: Cinder fails to create image-based volume if mirroring is enabled

[Vern Hart]

I tested Corey's fix (via ppa) side-by-side with a cinder unit with
version 2:15.4.0-0ubuntu1~cloud0 (from bionic-proposed/train).

I used a bionic qcow2 image with:
  openstack volume create --image bionic --size 3 --type stable2-az2 vernvol

Again, glance is backended by the az1 ceph cluster so this is a cross-
cluster copy. In our previous tests, qcow2 images tend to fail at a much
higher rate than raw images.

On the unpatched cinder, 77% failure with 23 out of 30 tries erring with:
  Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: deleting volume volume-1b4d5231-2baf-4880-a380-f1dc937b20bd that has snapshot

On Corey's patched version, 0% failure. 30 out of 30 volumes succeeded.

So, in this deployment, this simple retry solves the problem.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1900775

Title:
  Cinder fails to create image-based volume if mirroring is enabled

Status in charm-ceph-rbd-mirror:
  Invalid
Status in Cinder:
  In Progress
Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in Ubuntu Cloud Archive stein series:
  Triaged
Status in Ubuntu Cloud Archive train series:
  Triaged
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in cinder package in Ubuntu:
  Triaged
Status in cinder source package in Xenial:
  Triaged
Status in cinder source package in Bionic:
  Triaged
Status in cinder source package in Focal:
  Triaged
Status in cinder source package in Groovy:
  Triaged

Bug description:
  OpenStack Train, Ceph Nautilus, ceph-rbd-mirror is deployed in dual-way.
  Cinder has Ceph as backend for volumes.

  Creating a volume from qcow2 image.
  Current flow is the following:
  1. Cinder creates empty volume in Ceph
  2. Cinder downloads the image
  3. Image is converted to raw
  4. Volume is deleted https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1611
  5. Cinder performs "rbd import" using unpacked raw image as source.

  Apparently rbd-mirror daemon creates a snapshot upon the image creation in Ceph (seems for mirroring purposes) which for empty image lasts for about a second.
  It happens that step4 may be performed (very often) during the period of time when the snapshot exists, and it fails with "Cannot delete the volume with snapshots" error.
  The only way to fix this behaviour - disable mirroring of the backend pool which is not desired.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-rbd-mirror/+bug/1900775/+subscriptions