[Bug 1863021] Re: [SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError

[Corey Bryant]

** Also affects: networking-hyperv (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: networking-l2gw
   Importance: Undecided
       Status: New

** Also affects: networking-l2gw (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: networking-l2gw (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: networking-l2gw (Ubuntu Focal)
       Status: New => Triaged

** Changed in: networking-l2gw (Ubuntu Groovy)
   Importance: Undecided => High

** Changed in: networking-l2gw (Ubuntu Groovy)
       Status: New => Triaged

** Changed in: networking-hyperv (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: networking-hyperv (Ubuntu Focal)
       Status: New => Triaged

** Changed in: networking-hyperv (Ubuntu Groovy)
   Importance: Undecided => High

** Changed in: networking-hyperv (Ubuntu Groovy)
       Status: New => Triaged

** Also affects: networking-mlnx
   Importance: Undecided
       Status: New

** Also affects: networking-mlnx (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: networking-mlnx (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: networking-mlnx (Ubuntu Focal)
       Status: New => Triaged

** Changed in: networking-mlnx (Ubuntu Groovy)
   Importance: Undecided => High

** Changed in: networking-mlnx (Ubuntu Groovy)
       Status: New => Triaged

** Also affects: networking-sfc
   Importance: Undecided
       Status: New

** Also affects: networking-sfc (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: networking-sfc (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: networking-sfc (Ubuntu Focal)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to swift in Ubuntu.
https://bugs.launchpad.net/bugs/1863021

Title:
  [SRU] eventlet monkey patch results in assert len(_active) == 1
  AssertionError

Status in Cinder:
  In Progress
Status in Designate:
  In Progress
Status in Glance:
  Fix Released
Status in BaGPipe:
  In Progress
Status in networking-hyperv:
  In Progress
Status in networking-l2gw:
  In Progress
Status in Mellanox backend  integration with Neutron (networking-mlnx):
  In Progress
Status in networking-sfc:
  In Progress
Status in neutron:
  In Progress
Status in OpenStack Compute (nova):
  Fix Released
Status in oslo.service:
  In Progress
Status in OpenStack Object Storage (swift):
  In Progress
Status in barbican package in Ubuntu:
  Triaged
Status in cinder package in Ubuntu:
  Fix Released
Status in designate package in Ubuntu:
  Triaged
Status in glance package in Ubuntu:
  Fix Released
Status in heat package in Ubuntu:
  Triaged
Status in ironic package in Ubuntu:
  Triaged
Status in ironic-inspector package in Ubuntu:
  Triaged
Status in networking-bagpipe package in Ubuntu:
  Triaged
Status in networking-hyperv package in Ubuntu:
  Triaged
Status in networking-l2gw package in Ubuntu:
  Triaged
Status in networking-mlnx package in Ubuntu:
  Triaged
Status in networking-sfc package in Ubuntu:
  New
Status in neutron package in Ubuntu:
  Fix Released
Status in nova package in Ubuntu:
  Fix Released
Status in python-oslo.service package in Ubuntu:
  Triaged
Status in swift package in Ubuntu:
  Triaged
Status in barbican source package in Focal:
  Triaged
Status in cinder source package in Focal:
  Triaged
Status in designate source package in Focal:
  Triaged
Status in glance source package in Focal:
  Fix Released
Status in heat source package in Focal:
  Triaged
Status in ironic source package in Focal:
  Triaged
Status in ironic-inspector source package in Focal:
  Triaged
Status in networking-bagpipe source package in Focal:
  Triaged
Status in networking-hyperv source package in Focal:
  Triaged
Status in networking-l2gw source package in Focal:
  Triaged
Status in networking-mlnx source package in Focal:
  Triaged
Status in networking-sfc source package in Focal:
  Triaged
Status in neutron source package in Focal:
  Triaged
Status in nova source package in Focal:
  Fix Released
Status in python-oslo.service source package in Focal:
  Triaged
Status in swift source package in Focal:
  Triaged
Status in barbican source package in Groovy:
  Triaged
Status in cinder source package in Groovy:
  Fix Released
Status in designate source package in Groovy:
  Triaged
Status in glance source package in Groovy:
  Fix Released
Status in heat source package in Groovy:
  Triaged
Status in ironic source package in Groovy:
  Triaged
Status in ironic-inspector source package in Groovy:
  Triaged
Status in networking-bagpipe source package in Groovy:
  Triaged
Status in networking-hyperv source package in Groovy:
  Triaged
Status in networking-l2gw source package in Groovy:
  Triaged
Status in networking-mlnx source package in Groovy:
  Triaged
Status in networking-sfc source package in Groovy:
  New
Status in neutron source package in Groovy:
  Fix Released
Status in nova source package in Groovy:
  Fix Released
Status in python-oslo.service source package in Groovy:
  Triaged
Status in swift source package in Groovy:
  Triaged

Bug description:
  [Impact]
  This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592

  However I seem to only hit this with python3.8. Basically nova,
  glance, neutron, and cinder services fail with:

   Exception ignored in: <function _after_fork at 0x7f1100d075e0>
   Traceback (most recent call last):
     File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
       assert len(_active) == 1
   AssertionError:
   Exception ignored in: <function _after_fork at 0x7f1100d075e0>
   Traceback (most recent call last):
     File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
       assert len(_active) == 1
   AssertionError:

  Patching nova/monkey_patch.py with the following appears to fix this:

  diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py
  index a07ff91dac..bb7252c643 100644
  --- a/nova/monkey_patch.py
  +++ b/nova/monkey_patch.py
  @@ -59,6 +59,9 @@ def _monkey_patch():
       else:
           eventlet.monkey_patch()

  +    import __original_module_threading
  +    import threading
  +    __original_module_threading.current_thread.__globals__['_active'] = threading._active
       # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
       # hub use a monotonic clock to avoid issues with drifts of system time (see

  Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and
  glance/cmd/registry.py appears to fix it for glance.

  [Test Case]
  'systemctl status <service>' should not show the failure reported in [Impact].

  [Regression Potential]
  Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1863021/+subscriptions