[Bug 1838691] Re: Can't create incremental backups when using python3 and newer Ceph versions.

[James Page]

Ubuntu SRU information

[Impact]
Unable to create backups of ceph volumes via cinder.

[Test Case]
<Deploy OpenStack using charms>
openstack volume create --size 5 testvolume
openstack volume backup create testvolume
openstack volume backup create testvolume
openstack volume backup create testvolume

at least one backup will go into error status.

[Regression Potential]
Part of the 13.0.7 stable release from upstream openstack so low regression potential.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1838691

Title:
  Can't create incremental backups when using python3 and newer Ceph
  versions.

Status in Cinder:
  Fix Released
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Triaged
Status in Ubuntu Cloud Archive stein series:
  Triaged
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in cinder package in Ubuntu:
  Fix Released
Status in cinder source package in Disco:
  Triaged
Status in cinder source package in Eoan:
  Fix Released

Bug description:

  During the incremental backups creation, unexpectedly the
  '_cleanup_incomplete_backup_operations' method is called aborting the
  backup:

  DEBUG cinder.backup.drivers.ceph Volume file is RBD: attempting
  incremental backup. backup /usr/lib/python3.6/site-
  packages/cinder/backup/drivers/ceph.py:955

  INFO cinder.service [-] Starting cinder-backup node (version 14.0.1)

  DEBUG oslo_concurrency.lockutils Lock "cleanup_incomplete_backups_1"
  acquired by
  "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations"
  :: waited 0.001s inner /usr/lib/python3.6/site-
  packages/oslo_concurrency/lockutils.py:327

  INFO cinder.backup.manager Cleaning up incomplete backup operations.

  This happens when using ceph mimic with Python 3.x. The connections
  with Librbd[1] suddenly abort connections before accessing the data.
  [2]

  How to reproduce:

  >>> import rbd
  >>> import rados
  >>> cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
  >>> cluster.connect()
  >>> ioctx = cluster.open_ioctx('backups')
  >>> image = rbd.Image(ioctx, 'volume-2d2c563d-c567-431a-9b25-ea06af83c598.backup.base')
  >>> snaps = image.list_snaps()
  >>> image.close()
  >>> list(snaps)
  Aborted (core dumped)

  Expected output: snapshots list
  >>> list(snaps)
  [{'id': 4, 'size': 1073741824, 'name': 'backup.0dbaab38-595e-4e8a-8ecd-7ab60a879903.snap.1563912753.3520913', 'namespace': 0}, {'id': 5, 'size': 1073741824, 'name': 'backup.c9aaaf88-9ce1-4502-aa23-1dddfb1d45bb.snap.1564609930.8021717', 'namespace': 0}]

  Actual output: core dumped

  
  [1] https://docs.ceph.com/docs/giant/rbd/librbdpy/
  [2] https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/drivers/ceph.py#L649

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1838691/+subscriptions