[Bug 1863021] Re: [SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError

[Brian Murray]

Hello Corey, or anyone else affected,

Accepted masakari into focal-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/masakari/9.0.0-0ubuntu0.20.04.1 in
a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: masakari (Ubuntu Focal)
       Status: Triaged => Fix Committed

** Changed in: networking-mlnx (Ubuntu Focal)
       Status: Triaged => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to networking-hyperv in Ubuntu.
https://bugs.launchpad.net/bugs/1863021

Title:
  [SRU] eventlet monkey patch results in assert len(_active) == 1
  AssertionError

Status in Cinder:
  Fix Released
Status in Designate:
  In Progress
Status in Glance:
  Fix Released
Status in OpenStack Shared File Systems Service (Manila):
  Fix Released
Status in masakari:
  In Progress
Status in Mistral:
  Fix Released
Status in Murano:
  Fix Released
Status in BaGPipe:
  Fix Released
Status in networking-hyperv:
  Fix Released
Status in networking-l2gw:
  Fix Released
Status in Mellanox backend  integration with Neutron (networking-mlnx):
  In Progress
Status in networking-sfc:
  Fix Released
Status in neutron:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in oslo.service:
  Fix Released
Status in senlin:
  Fix Released
Status in OpenStack Object Storage (swift):
  In Progress
Status in OpenStack DBaaS (Trove):
  In Progress
Status in watcher:
  Fix Released
Status in barbican package in Ubuntu:
  Fix Released
Status in cinder package in Ubuntu:
  Fix Released
Status in designate package in Ubuntu:
  Fix Released
Status in glance package in Ubuntu:
  Fix Released
Status in heat package in Ubuntu:
  Triaged
Status in ironic package in Ubuntu:
  Fix Released
Status in ironic-inspector package in Ubuntu:
  Fix Released
Status in magnum package in Ubuntu:
  Fix Released
Status in manila package in Ubuntu:
  Fix Released
Status in masakari package in Ubuntu:
  Triaged
Status in mistral package in Ubuntu:
  Fix Released
Status in murano package in Ubuntu:
  Fix Released
Status in murano-agent package in Ubuntu:
  Fix Released
Status in networking-bagpipe package in Ubuntu:
  Fix Released
Status in networking-hyperv package in Ubuntu:
  Fix Released
Status in networking-l2gw package in Ubuntu:
  Fix Released
Status in networking-mlnx package in Ubuntu:
  Fix Released
Status in networking-sfc package in Ubuntu:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron-dynamic-routing package in Ubuntu:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in openstack-trove package in Ubuntu:
  Fix Released
Status in python-os-ken package in Ubuntu:
  Fix Released
Status in python-oslo.service package in Ubuntu:
  Fix Released
Status in sahara package in Ubuntu:
  Triaged
Status in senlin package in Ubuntu:
  Fix Released
Status in swift package in Ubuntu:
  Triaged
Status in watcher package in Ubuntu:
  Fix Released
Status in barbican source package in Focal:
  Fix Committed
Status in cinder source package in Focal:
  Triaged
Status in designate source package in Focal:
  Fix Committed
Status in glance source package in Focal:
  Fix Released
Status in heat source package in Focal:
  Fix Committed
Status in ironic source package in Focal:
  Fix Committed
Status in ironic-inspector source package in Focal:
  Fix Committed
Status in magnum source package in Focal:
  Fix Committed
Status in manila source package in Focal:
  Triaged
Status in masakari source package in Focal:
  Fix Committed
Status in mistral source package in Focal:
  Triaged
Status in murano source package in Focal:
  Triaged
Status in murano-agent source package in Focal:
  Triaged
Status in networking-bagpipe source package in Focal:
  Triaged
Status in networking-hyperv source package in Focal:
  Triaged
Status in networking-l2gw source package in Focal:
  Triaged
Status in networking-mlnx source package in Focal:
  Fix Committed
Status in networking-sfc source package in Focal:
  Triaged
Status in neutron source package in Focal:
  Triaged
Status in neutron-dynamic-routing source package in Focal:
  Triaged
Status in nova source package in Focal:
  Fix Released
Status in openstack-trove source package in Focal:
  Fix Committed
Status in python-os-ken source package in Focal:
  Triaged
Status in python-oslo.service source package in Focal:
  Triaged
Status in sahara source package in Focal:
  Triaged
Status in senlin source package in Focal:
  Triaged
Status in swift source package in Focal:
  Triaged
Status in watcher source package in Focal:
  Triaged
Status in barbican source package in Groovy:
  Fix Released
Status in cinder source package in Groovy:
  Fix Released
Status in designate source package in Groovy:
  Fix Released
Status in glance source package in Groovy:
  Fix Released
Status in heat source package in Groovy:
  Triaged
Status in ironic source package in Groovy:
  Fix Released
Status in ironic-inspector source package in Groovy:
  Fix Released
Status in magnum source package in Groovy:
  Fix Released
Status in manila source package in Groovy:
  Fix Released
Status in masakari source package in Groovy:
  Triaged
Status in mistral source package in Groovy:
  Fix Released
Status in murano source package in Groovy:
  Fix Released
Status in murano-agent source package in Groovy:
  Fix Released
Status in networking-bagpipe source package in Groovy:
  Fix Released
Status in networking-hyperv source package in Groovy:
  Fix Released
Status in networking-l2gw source package in Groovy:
  Fix Released
Status in networking-mlnx source package in Groovy:
  Fix Released
Status in networking-sfc source package in Groovy:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron-dynamic-routing source package in Groovy:
  Fix Released
Status in nova source package in Groovy:
  Fix Released
Status in openstack-trove source package in Groovy:
  Fix Released
Status in python-os-ken source package in Groovy:
  Fix Released
Status in python-oslo.service source package in Groovy:
  Fix Released
Status in sahara source package in Groovy:
  Triaged
Status in senlin source package in Groovy:
  Fix Released
Status in swift source package in Groovy:
  Triaged
Status in watcher source package in Groovy:
  Fix Released

Bug description:
  [Impact]
  This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592

  However I seem to only hit this with python3.8. Basically nova,
  glance, neutron, and cinder services fail with:

   Exception ignored in: <function _after_fork at 0x7f1100d075e0>
   Traceback (most recent call last):
     File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
       assert len(_active) == 1
   AssertionError:
   Exception ignored in: <function _after_fork at 0x7f1100d075e0>
   Traceback (most recent call last):
     File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
       assert len(_active) == 1
   AssertionError:

  Patching nova/monkey_patch.py with the following appears to fix this:

  diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py
  index a07ff91dac..bb7252c643 100644
  --- a/nova/monkey_patch.py
  +++ b/nova/monkey_patch.py
  @@ -59,6 +59,9 @@ def _monkey_patch():
       else:
           eventlet.monkey_patch()

  +    import __original_module_threading
  +    import threading
  +    __original_module_threading.current_thread.__globals__['_active'] = threading._active
       # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
       # hub use a monotonic clock to avoid issues with drifts of system time (see

  Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and
  glance/cmd/registry.py appears to fix it for glance.

  [Test Case]
  Tempest regression on a deployed openstack should pass successfully.

  Individual services can be tested as follows but with the scope of this issue tempest regression testing will be the goal for verifying this SRU.
  'systemctl status <service>' should not show the failure reported in 

  [Impact].

  [Regression Potential]
  Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1863021/+subscriptions