[Bug 1834845] Re: DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session

OpenStack Infra 1834845 at bugs.launchpad.net
Thu Aug 8 13:58:09 UTC 2019 

Reviewed:  https://review.opendev.org/675081
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Submitter: Zuul
Branch:    stable/stein

commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Author: Gorka Eguileor <geguileo at redhat.com>
Date:   Tue Jul 2 11:06:17 2019 +0200

    Fix DetachedInstanceError for VolumeAttachment
    
    Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
    DetachedInstanceError for VolumeAttachment OVOs but only partially, as
    apparently it was dependent on the SQLAlchemy version due to the use os
    "hasattr".
    
    This patch replaces "hasattr" with a check on the object's dictionary,
    which will never trigger a Lazy Load.
    
    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
    (cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)


** Changed in: cloud-archive/stein
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1834845

Title:
  DetachedInstanceError: Parent instance <VolumeAttachment at > is not
  bound to a Session

Status in Cinder:
  Fix Released
Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  In Progress
Status in cinder package in Ubuntu:
  Fix Released
Status in cinder source package in Disco:
  In Progress
Status in cinder source package in Eoan:
  Fix Released

Bug description:
  Hey there.

  We upgraded from rocky to stein and facing a strange error. Our
  cinder-ceph volume services gets marked as down after a few seconds.
  So we looked into this and found a error in the logs:

  Error starting thread.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service Traceback (most recent call last):
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service.start()
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 222, in start
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service_id=Service.service_id)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 445, in init_host
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 523, in _init_host
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes = self._get_my_volumes(ctxt)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3007, in _get_my_volumes
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service limit, offset)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3003, in _get_my_resources
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service offset=offset)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 617, in get_all
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes, expected_attrs=expected_attrs)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 290, in _from_db_object
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service db_volume.get('volume_attachment'))
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume_attachment.py", line 102, in _from_db_object
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service if 'volume' in expected_attrs and hasattr(db_attachment, 'volume'):
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 242, in __get__
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service return self.impl.get(instance_state(instance), dict_)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 601, in get
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service value = self.callable_(state, passive)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/strategies.py", line 596, in _load_for_state
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service (orm_util.state_str(state), self.key)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

  It looks like, because of this error the service gets marked down. This issue is definitly linked to this: https://bugs.launchpad.net/cinder/+bug/1812913 but the fix that is released is not fixing it.
  If we put a try-catch block around this block, the services stays up, but i am not sure if something else will break.
  This is in our perspective a high priority bug, because we are not able to use one of our ceph backends.

  ii cinder-api 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - API server
  ii cinder-common 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - common files
  ii cinder-scheduler 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
  ii cinder-volume 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Volume server
  ii python3-cinder 2:14.0.0-0ubuntu1~cloud0 all Cinder Python 3 libraries

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1834845/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list