[Bug 1834845] Re: DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session
OpenStack Infra
1834845 at bugs.launchpad.net
Thu Aug 8 13:58:09 UTC 2019
Reviewed: https://review.opendev.org/675081
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Submitter: Zuul
Branch: stable/stein
commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Author: Gorka Eguileor <geguileo at redhat.com>
Date: Tue Jul 2 11:06:17 2019 +0200
Fix DetachedInstanceError for VolumeAttachment
Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
DetachedInstanceError for VolumeAttachment OVOs but only partially, as
apparently it was dependent on the SQLAlchemy version due to the use os
"hasattr".
This patch replaces "hasattr" with a check on the object's dictionary,
which will never trigger a Lazy Load.
Closes-Bug: #1834845
Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
(cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)
** Changed in: cloud-archive/stein
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1834845
Title:
DetachedInstanceError: Parent instance <VolumeAttachment at > is not
bound to a Session
Status in Cinder:
Fix Released
Status in Ubuntu Cloud Archive:
Fix Committed
Status in Ubuntu Cloud Archive stein series:
Fix Committed
Status in Ubuntu Cloud Archive train series:
In Progress
Status in cinder package in Ubuntu:
Fix Released
Status in cinder source package in Disco:
In Progress
Status in cinder source package in Eoan:
Fix Released
Bug description:
Hey there.
We upgraded from rocky to stein and facing a strange error. Our
cinder-ceph volume services gets marked as down after a few seconds.
So we looked into this and found a error in the logs:
Error starting thread.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service Traceback (most recent call last):
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service.start()
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 222, in start
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service_id=Service.service_id)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 445, in init_host
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 523, in _init_host
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes = self._get_my_volumes(ctxt)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3007, in _get_my_volumes
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service limit, offset)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3003, in _get_my_resources
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service offset=offset)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 617, in get_all
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes, expected_attrs=expected_attrs)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 290, in _from_db_object
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service db_volume.get('volume_attachment'))
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume_attachment.py", line 102, in _from_db_object
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service if 'volume' in expected_attrs and hasattr(db_attachment, 'volume'):
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 242, in __get__
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service return self.impl.get(instance_state(instance), dict_)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 601, in get
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service value = self.callable_(state, passive)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/strategies.py", line 596, in _load_for_state
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service (orm_util.state_str(state), self.key)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
It looks like, because of this error the service gets marked down. This issue is definitly linked to this: https://bugs.launchpad.net/cinder/+bug/1812913 but the fix that is released is not fixing it.
If we put a try-catch block around this block, the services stays up, but i am not sure if something else will break.
This is in our perspective a high priority bug, because we are not able to use one of our ceph backends.
ii cinder-api 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - API server
ii cinder-common 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - common files
ii cinder-scheduler 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Volume server
ii python3-cinder 2:14.0.0-0ubuntu1~cloud0 all Cinder Python 3 libraries
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1834845/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list