[Bug 1869746] Re: Cinder throws error creating incremental backup from parent in another project
Rodrigo Barbieri
1869746 at bugs.launchpad.net
Fri Jul 24 16:08:42 UTC 2020
** Description changed:
Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver
Ceph
This bug was also reproduced on Queens.
Steps to reproduce:
1) Create a volume v1 as user demo
2) Create a full backup of v1 as user admin
3) Create an incremental backup of v1 as user demo
4) Result: Backup is created successfully, but error in logs. Logs do not show INFO message and parent_backup.num_dependent_backups is not incremented [0].
Expectation: Either no error successfully finishing method
create_backup, or not creating backup at all, with backup entity set as
Error (see below).
Using POSIX driver, the backup is not created successfully and backup
entity remains in error state. The POSIX driver tries to read
backup.parent_id before creating the backup therefore raises the error
earlier.
Trace:
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5 9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9 f883c6315a004471b2ab1b04d8a111d9] Exception during message handling: cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found.
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in create_backup
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server backup.parent_id)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in backup_get
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server project_only=project_only)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in _backup_get
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise exception.BackupNotFound(backup_id=backup_id)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found.
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
[0]
https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448
+
+ ====================================================================
+
+ [Impact]
+
+ Whenever an admin creates a backup for a tenant volume, all subsequent
+ incremental backups will have a silent error during creation when
+ linking the dependent backup to its parent because the code detects it
+ has a parent (by listing backups from all projects) but is unable to
+ find the DB entity while scoped to only its project. Upon deleting the
+ child backup, it will hit another silent error (because of lack of link)
+ and it will skip cleaning up the quota for the backup. The quota will
+ then misrepresent the number and size of backups, and the discrepancy
+ will increase as this error is hit for several backups. Ultimately, the
+ user will hit a quota error with a lower number/size of backups than the
+ actual quota limit.
+
+ The fix addressed the problem by changing the code that searches for the
+ parent backup. The code will now only search for backups under the same
+ project, so it will prevent the error from happening.
+
+ [Test case]
+
+ 1) Reproducing the bug:
+
+ 1a) Create a volume v1 as user demo
+ 1b) Create a full backup b1 of v1 as user admin
+ 1c) Create an incremental backup b2 of v1 as user demo
+ 1d) Confirm backup b2 is created successfully
+ 1e) Check log file "/var/log/cinder/cinder-backup.log" for error message below with b2's ID.
+
+ cinder.exception.BackupNotFound: Backup <b2_id> could not be found
+
+ 2) Cleanup not necessary
+
+ 3) Install package that contains fixed code
+
+ 4) Confirm bug is fixed
+
+ 4a) Repeat steps 1a to 1d, make sure to create a new volume and new backups from it.
+ 4b) Check log file "/var/log/cinder/cinder-backup.log" for error messages related to new b2 ID. There shouldn't be any.
+
+ [Regression Potential]
+
+ The fix has already been validated in upstream CI with new functional
+ tests that perform that specific workflow (however, checks API backup
+ field instead of logs), as far back as Rocky release. The code changes
+ affect the creation step of backups, and both the new and previously
+ existing functional tests thoroughly test the creation step in different
+ ways, therefore regressions with this patch are not expected.
+
+ [Other Info]
+
+ For releases older than Train, the fix for bug 1809323 needs to be
+ included as a dependency.
** Also affects: cinder (Ubuntu)
Importance: Undecided
Status: New
** Also affects: cloud-archive
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1869746
Title:
Cinder throws error creating incremental backup from parent in another
project
Status in Cinder:
Fix Released
Status in Ubuntu Cloud Archive:
New
Status in cinder package in Ubuntu:
New
Bug description:
Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver
Ceph
This bug was also reproduced on Queens.
Steps to reproduce:
1) Create a volume v1 as user demo
2) Create a full backup of v1 as user admin
3) Create an incremental backup of v1 as user demo
4) Result: Backup is created successfully, but error in logs. Logs do not show INFO message and parent_backup.num_dependent_backups is not incremented [0].
Expectation: Either no error successfully finishing method
create_backup, or not creating backup at all, with backup entity set
as Error (see below).
Using POSIX driver, the backup is not created successfully and backup
entity remains in error state. The POSIX driver tries to read
backup.parent_id before creating the backup therefore raises the error
earlier.
Trace:
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5 9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9 f883c6315a004471b2ab1b04d8a111d9] Exception during message handling: cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found.
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in create_backup
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server backup.parent_id)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in get_by_id
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in backup_get
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server project_only=project_only)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in _backup_get
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise exception.BackupNotFound(backup_id=backup_id)
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found.
2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server
[0]
https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448
====================================================================
[Impact]
Whenever an admin creates a backup for a tenant volume, all subsequent
incremental backups will have a silent error during creation when
linking the dependent backup to its parent because the code detects it
has a parent (by listing backups from all projects) but is unable to
find the DB entity while scoped to only its project. Upon deleting the
child backup, it will hit another silent error (because of lack of
link) and it will skip cleaning up the quota for the backup. The quota
will then misrepresent the number and size of backups, and the
discrepancy will increase as this error is hit for several backups.
Ultimately, the user will hit a quota error with a lower number/size
of backups than the actual quota limit.
The fix addressed the problem by changing the code that searches for
the parent backup. The code will now only search for backups under the
same project, so it will prevent the error from happening.
[Test case]
1) Reproducing the bug:
1a) Create a volume v1 as user demo
1b) Create a full backup b1 of v1 as user admin
1c) Create an incremental backup b2 of v1 as user demo
1d) Confirm backup b2 is created successfully
1e) Check log file "/var/log/cinder/cinder-backup.log" for error message below with b2's ID.
cinder.exception.BackupNotFound: Backup <b2_id> could not be found
2) Cleanup not necessary
3) Install package that contains fixed code
4) Confirm bug is fixed
4a) Repeat steps 1a to 1d, make sure to create a new volume and new backups from it.
4b) Check log file "/var/log/cinder/cinder-backup.log" for error messages related to new b2 ID. There shouldn't be any.
[Regression Potential]
The fix has already been validated in upstream CI with new functional
tests that perform that specific workflow (however, checks API backup
field instead of logs), as far back as Rocky release. The code changes
affect the creation step of backups, and both the new and previously
existing functional tests thoroughly test the creation step in
different ways, therefore regressions with this patch are not
expected.
[Other Info]
For releases older than Train, the fix for bug 1809323 needs to be
included as a dependency.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1869746/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list