[Bug 1838691] Re: Can't create incremental backups when using python3 and newer Ceph versions.

[Corey Bryant]

Verified successfully on stein-proposed:

$ juju ssh cinder/0 apt policy cinder-common
cinder-common:
  Installed: 2:14.0.2-0ubuntu1~cloud0
  Candidate: 2:14.0.2-0ubuntu1~cloud0
  Version table:
 *** 2:14.0.2-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64 Packages

$ openstack volume list
+--------------------------------------+------+-----------+------+-----------------------------+
| ID                                   | Name | Status    | Size | Attached to                 |
+--------------------------------------+------+-----------+------+-----------------------------+
| 41b29ac2-c580-444c-ba65-17d016fb136f | v2   | available |    5 |                             |
| 760b2fe5-bfa3-4ec6-964d-6fd05ccba3b0 | v1   | in-use    |    1 | Attached to x1 on /dev/vdb  |
+--------------------------------------+------+-----------+------+-----------------------------+

$ openstack volume backup create v2
$ openstack volume backup create v2
$ openstack volume backup create v2

$ openstack volume backup list
+--------------------------------------+------+-------------+-----------+------+
| ID                                   | Name | Description | Status    | Size |
+--------------------------------------+------+-------------+-----------+------+
| 2d2d2aa8-f3ef-4b1b-b3a4-8f187369e632 | None | None        | available |    5 |
| 5892fc21-ac63-4e78-979d-9145b8c67db2 | None | None        | available |    5 |
| 1bf0f9ab-b6b7-4cca-a46a-55853ec7f204 | None | None        | available |    5 |
+--------------------------------------+------+-------------+-----------+------+

** Tags removed: verification-stein-needed
** Tags added: verification-stein-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1838691

Title:
  Can't create incremental backups when using python3 and newer Ceph
  versions.

Status in Cinder:
  Fix Released
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in cinder package in Ubuntu:
  Fix Released
Status in cinder source package in Disco:
  Won't Fix
Status in cinder source package in Eoan:
  Fix Released

Bug description:

  During the incremental backups creation, unexpectedly the
  '_cleanup_incomplete_backup_operations' method is called aborting the
  backup:

  DEBUG cinder.backup.drivers.ceph Volume file is RBD: attempting
  incremental backup. backup /usr/lib/python3.6/site-
  packages/cinder/backup/drivers/ceph.py:955

  INFO cinder.service [-] Starting cinder-backup node (version 14.0.1)

  DEBUG oslo_concurrency.lockutils Lock "cleanup_incomplete_backups_1"
  acquired by
  "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations"
  :: waited 0.001s inner /usr/lib/python3.6/site-
  packages/oslo_concurrency/lockutils.py:327

  INFO cinder.backup.manager Cleaning up incomplete backup operations.

  This happens when using ceph mimic with Python 3.x. The connections
  with Librbd[1] suddenly abort connections before accessing the data.
  [2]

  How to reproduce:

  >>> import rbd
  >>> import rados
  >>> cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
  >>> cluster.connect()
  >>> ioctx = cluster.open_ioctx('backups')
  >>> image = rbd.Image(ioctx, 'volume-2d2c563d-c567-431a-9b25-ea06af83c598.backup.base')
  >>> snaps = image.list_snaps()
  >>> image.close()
  >>> list(snaps)
  Aborted (core dumped)

  Expected output: snapshots list
  >>> list(snaps)
  [{'id': 4, 'size': 1073741824, 'name': 'backup.0dbaab38-595e-4e8a-8ecd-7ab60a879903.snap.1563912753.3520913', 'namespace': 0}, {'id': 5, 'size': 1073741824, 'name': 'backup.c9aaaf88-9ce1-4502-aa23-1dddfb1d45bb.snap.1564609930.8021717', 'namespace': 0}]

  Actual output: core dumped

  
  [1] https://docs.ceph.com/docs/giant/rbd/librbdpy/
  [2] https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/drivers/ceph.py#L649

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1838691/+subscriptions