[Bug 1831735] Re: Google backup driver fails to backup a volume

Mon Apr 27 17:52:00 UTC 2020

** Also affects: cinder (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1831735

Title:
  Google backup driver fails to backup a volume

Status in Cinder:
  Fix Released
Status in cinder package in Ubuntu:
  New

Bug description:
  Description of problem: Cinder volume backup using Google as backup
  backend fails. We suspect this regression was introduced by move to
  Python3.

  Version-Release number of selected component (if applicable):
  rhel8
  puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch                                                                                                                                                                                  
  openstack-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                               
  python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch                                                                                                                                                                            
  python3-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                                 

  python3-google-api-client-1.6.5-3.el8.noarch

  How reproducible:
  Every time

  Steps to Reproduce:
  1. Configure Google backup.

  Cinder backup service is enabled and up
  $ cinder service-list
  +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
  | Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
  +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
  | cinder-backup    | controller-0            | nova | enabled | up    | 2019-06-05T11:17:47.000000 | -               |

  2. Create a volume 
  vol id -> 0336a629-25c4-4e05-be06-de51a75afc9d

  3. If you try to backup the volume, backup fails
  cinder backup logs reports the trace show below

  $ cinder backup-list
  +--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
  | ID                                   | Volume ID                            | Status | Name | Size | Object Count | Container      |
  +--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
  | d256989b-a88f-4dcf-95d2-1c7afa12be76 | 0336a629-25c4-4e05-be06-de51a75afc9d | error  | -    | 1    | 0            | mycinderbucket |
  +--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+

  e trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:170
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server [req-4f591f5c-031a-417e-974c-37cdb2ced102 c45b531877804ab097b8f587497031db ac390c9999244f56aa05cb13e181f0bc - default default] Exception during message handling: cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 121, in func_wrapper
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 312, in close
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     md5 = md5.encode('utf-8')
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 406, in create_backup
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self._update_backup_error(backup, six.text_type(err))
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 395, in create_backup
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 479, in _run_backup
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     backup, tpool.Proxy(device_file))
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 625, in backup
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     object_meta, extra_metadata)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 374, in _backup_chunk
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     writer.write(output_data)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 221, in __exit__
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return proxy_call(self._autowrap, self._obj.__exit__, *exc)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 293, in __exit__
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.close()
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 127, in func_wrapper
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise exception.GCSConnectionFailure(reason=err)
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
  2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
  2019-06-05 09:25:57.656 21 DEBUG oslo_service.periodic_task [req-dbe4b355-7960-4100-a621-4fba6a5c05ac - - - - -] Running periodic task BackupManager._report_driver_status run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217

  Actual results:
  Cinder backup fails 

  Expected results:
  Cinder backup should successfully complete the backup.
  Least this worked on previous openstack release.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1831735/+subscriptions