[Bug 1863021] Fix merged to oslo.service (stable/ussuri)
OpenStack Infra
1863021 at bugs.launchpad.net
Thu May 7 16:58:44 UTC 2020
Reviewed: https://review.opendev.org/725613
Committed: https://git.openstack.org/cgit/openstack/oslo.service/commit/?id=b52454d1d07701a49d14b4c1b52b5d7cec10084e
Submitter: Zuul
Branch: stable/ussuri
commit b52454d1d07701a49d14b4c1b52b5d7cec10084e
Author: Corey Bryant <corey.bryant at canonical.com>
Date: Mon May 4 14:06:38 2020 -0400
Monkey patch original current_thread _active
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: Icc2277b72f6f8f7812be22c43bbc281334aa2373
Closes-Bug: #1863021
(cherry picked from commit b83ce338312bfeabda6b6755f49530acde61141e)
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to networking-hyperv in Ubuntu.
https://bugs.launchpad.net/bugs/1863021
Title:
[SRU] eventlet monkey patch results in assert len(_active) == 1
AssertionError
Status in Cinder:
Fix Released
Status in Designate:
In Progress
Status in Glance:
Fix Released
Status in OpenStack Shared File Systems Service (Manila):
In Progress
Status in masakari:
In Progress
Status in Mistral:
In Progress
Status in Murano:
Fix Released
Status in BaGPipe:
Fix Released
Status in networking-hyperv:
Fix Released
Status in networking-l2gw:
Fix Released
Status in Mellanox backend integration with Neutron (networking-mlnx):
In Progress
Status in networking-sfc:
Fix Released
Status in neutron:
Fix Released
Status in OpenStack Compute (nova):
Fix Released
Status in oslo.service:
Fix Released
Status in senlin:
In Progress
Status in OpenStack Object Storage (swift):
In Progress
Status in OpenStack DBaaS (Trove):
In Progress
Status in watcher:
In Progress
Status in barbican package in Ubuntu:
Triaged
Status in cinder package in Ubuntu:
Fix Released
Status in designate package in Ubuntu:
Triaged
Status in glance package in Ubuntu:
Fix Released
Status in heat package in Ubuntu:
Triaged
Status in ironic package in Ubuntu:
Triaged
Status in ironic-inspector package in Ubuntu:
Triaged
Status in magnum package in Ubuntu:
Triaged
Status in manila package in Ubuntu:
Triaged
Status in masakari package in Ubuntu:
Triaged
Status in mistral package in Ubuntu:
Triaged
Status in murano package in Ubuntu:
Triaged
Status in murano-agent package in Ubuntu:
Triaged
Status in networking-bagpipe package in Ubuntu:
Triaged
Status in networking-hyperv package in Ubuntu:
Triaged
Status in networking-l2gw package in Ubuntu:
Triaged
Status in networking-mlnx package in Ubuntu:
Triaged
Status in networking-sfc package in Ubuntu:
Triaged
Status in neutron package in Ubuntu:
Fix Released
Status in neutron-dynamic-routing package in Ubuntu:
Triaged
Status in nova package in Ubuntu:
Fix Released
Status in openstack-trove package in Ubuntu:
Triaged
Status in python-os-ken package in Ubuntu:
Triaged
Status in python-oslo.service package in Ubuntu:
Triaged
Status in sahara package in Ubuntu:
Triaged
Status in senlin package in Ubuntu:
Triaged
Status in swift package in Ubuntu:
Triaged
Status in watcher package in Ubuntu:
Triaged
Status in barbican source package in Focal:
Triaged
Status in cinder source package in Focal:
Triaged
Status in designate source package in Focal:
Triaged
Status in glance source package in Focal:
Fix Released
Status in heat source package in Focal:
Triaged
Status in ironic source package in Focal:
Triaged
Status in ironic-inspector source package in Focal:
Triaged
Status in magnum source package in Focal:
Triaged
Status in manila source package in Focal:
Triaged
Status in masakari source package in Focal:
Triaged
Status in mistral source package in Focal:
Triaged
Status in murano source package in Focal:
Triaged
Status in murano-agent source package in Focal:
Triaged
Status in networking-bagpipe source package in Focal:
Triaged
Status in networking-hyperv source package in Focal:
Triaged
Status in networking-l2gw source package in Focal:
Triaged
Status in networking-mlnx source package in Focal:
Triaged
Status in networking-sfc source package in Focal:
Triaged
Status in neutron source package in Focal:
Triaged
Status in neutron-dynamic-routing source package in Focal:
Triaged
Status in nova source package in Focal:
Fix Released
Status in openstack-trove source package in Focal:
Triaged
Status in python-os-ken source package in Focal:
Triaged
Status in python-oslo.service source package in Focal:
Triaged
Status in sahara source package in Focal:
Triaged
Status in senlin source package in Focal:
Triaged
Status in swift source package in Focal:
Triaged
Status in watcher source package in Focal:
Triaged
Status in barbican source package in Groovy:
Triaged
Status in cinder source package in Groovy:
Fix Released
Status in designate source package in Groovy:
Triaged
Status in glance source package in Groovy:
Fix Released
Status in heat source package in Groovy:
Triaged
Status in ironic source package in Groovy:
Triaged
Status in ironic-inspector source package in Groovy:
Triaged
Status in magnum source package in Groovy:
Triaged
Status in manila source package in Groovy:
Triaged
Status in masakari source package in Groovy:
Triaged
Status in mistral source package in Groovy:
Triaged
Status in murano source package in Groovy:
Triaged
Status in murano-agent source package in Groovy:
Triaged
Status in networking-bagpipe source package in Groovy:
Triaged
Status in networking-hyperv source package in Groovy:
Triaged
Status in networking-l2gw source package in Groovy:
Triaged
Status in networking-mlnx source package in Groovy:
Triaged
Status in networking-sfc source package in Groovy:
Triaged
Status in neutron source package in Groovy:
Fix Released
Status in neutron-dynamic-routing source package in Groovy:
Triaged
Status in nova source package in Groovy:
Fix Released
Status in openstack-trove source package in Groovy:
Triaged
Status in python-os-ken source package in Groovy:
Triaged
Status in python-oslo.service source package in Groovy:
Triaged
Status in sahara source package in Groovy:
Triaged
Status in senlin source package in Groovy:
Triaged
Status in swift source package in Groovy:
Triaged
Status in watcher source package in Groovy:
Triaged
Bug description:
[Impact]
This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592
However I seem to only hit this with python3.8. Basically nova,
glance, neutron, and cinder services fail with:
Exception ignored in: <function _after_fork at 0x7f1100d075e0>
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
assert len(_active) == 1
AssertionError:
Exception ignored in: <function _after_fork at 0x7f1100d075e0>
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
assert len(_active) == 1
AssertionError:
Patching nova/monkey_patch.py with the following appears to fix this:
diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py
index a07ff91dac..bb7252c643 100644
--- a/nova/monkey_patch.py
+++ b/nova/monkey_patch.py
@@ -59,6 +59,9 @@ def _monkey_patch():
else:
eventlet.monkey_patch()
+ import __original_module_threading
+ import threading
+ __original_module_threading.current_thread.__globals__['_active'] = threading._active
# NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet
# hub use a monotonic clock to avoid issues with drifts of system time (see
Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and
glance/cmd/registry.py appears to fix it for glance.
[Test Case]
'systemctl status <service>' should not show the failure reported in [Impact].
[Regression Potential]
Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1863021/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list