[Bug 1834845] Re: DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session

[Eric Harney]

** Changed in: cinder
     Assignee: Eric Harney (eharney) => Gorka Eguileor (gorka)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Ubuntu.
https://bugs.launchpad.net/bugs/1834845

Title:
  DetachedInstanceError: Parent instance <VolumeAttachment at > is not
  bound to a Session

Status in Cinder:
  In Progress
Status in cinder package in Ubuntu:
  New

Bug description:
  Hey there.

  We upgraded from rocky to stein and facing a strange error. Our
  cinder-ceph volume services gets marked as down after a few seconds.
  So we looked into this and found a error in the logs:

  Error starting thread.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service Traceback (most recent call last):
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service.start()
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 222, in start
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service_id=Service.service_id)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 445, in init_host
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 523, in _init_host
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes = self._get_my_volumes(ctxt)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3007, in _get_my_volumes
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service limit, offset)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3003, in _get_my_resources
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service offset=offset)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 617, in get_all
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes, expected_attrs=expected_attrs)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 290, in _from_db_object
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service db_volume.get('volume_attachment'))
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume_attachment.py", line 102, in _from_db_object
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service if 'volume' in expected_attrs and hasattr(db_attachment, 'volume'):
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 242, in __get__
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service return self.impl.get(instance_state(instance), dict_)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 601, in get
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service value = self.callable_(state, passive)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/strategies.py", line 596, in _load_for_state
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service (orm_util.state_str(state), self.key)
  2019-07-01 10:29:22.947 16745 ERROR oslo_service.service sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

  It looks like, because of this error the service gets marked down. This issue is definitly linked to this: https://bugs.launchpad.net/cinder/+bug/1812913 but the fix that is released is not fixing it.
  If we put a try-catch block around this block, the services stays up, but i am not sure if something else will break.
  This is in our perspective a high priority bug, because we are not able to use one of our ceph backends.

  ii cinder-api 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - API server
  ii cinder-common 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - common files
  ii cinder-scheduler 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
  ii cinder-volume 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Volume server
  ii python3-cinder 2:14.0.0-0ubuntu1~cloud0 all Cinder Python 3 libraries

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1834845/+subscriptions