[Bug 2018405] Re: neutron ovsdbapp ssl connection stuck in OSError error loop
Edward Hope-Morley
2018405 at bugs.launchpad.net
Thu May 4 10:58:01 UTC 2023
** Patch added: "lp2018405-mantic.debdiff"
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2018405/+attachment/5670908/+files/lp2018405-mantic.debdiff
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/2018405
Title:
[SRU] neutron ovsdbapp ssl connection stuck in OSError error loop
Status in openvswitch package in Ubuntu:
New
Status in openvswitch source package in Jammy:
New
Status in openvswitch source package in Kinetic:
New
Status in openvswitch source package in Lunar:
New
Status in openvswitch source package in Mantic:
New
Bug description:
[Impact]
terminated/closed ssl connections between openstack agents and the ovn-central databases are not properly handled and this patch fixes that.
[Test Plan]
* deploy Openstack with OVN
* abruptly terminate connections between neutron-server and/or neutron-ovn-metadata-agent and ovn-central NB/SB database servers.
* ensure that connections are re-established and the errors below no not appear.
[Regression Potential]
No regressions are anticipated as a result of using this patch.
---------------------------------------------------------------------
We are running Openstack Yoga on both Ubuntu Focal and Ubuntu Jammy
and both environments are experiencing the following errors in
neutron-server and neutron-ovn-metadata-agent:
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection [req-302f6be6-5b72-49b4-9132-d9ec39370fd5 - - - - -] [Errno 107] Transport endpoint is not connected: OSError: [Errno 107] Transport endpoint is not connected
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 108, in run
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection self.idl.run()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection self._session.run()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection error = self.stream.connect()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection self.socket.do_handshake()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection return self._call_trampolining(
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection return func(*a, **kw)
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3.10/ssl.py", line 1337, in do_handshake
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection self._check_connected()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3.10/ssl.py", line 1119, in _check_connected
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection self.getpeername()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection OSError: [Errno 107] Transport endpoint is not connected
This reconnect error happens forever after an initial ssl disconnect
like:
2023-04-12 19:15:40.281 2146189 ERROR ovsdbapp.backend.ovs_idl.transaction [req-7d895085-c4e1-43b9-a36f-808e86ef5caa f3797dd627d24377a7d0a1330595aed9 e4b04ca58d734ed0aa29e306adad4f79 - 93f9c955065f4cb38b9a8f1b98eedb92 93f9c955065f4cb38b9a8f1b98eedb92] Traceback (most recent
call last):
File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
txn.results.put(txn.do_commit())
File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 114, in do_commit
self.api.idl.run()
File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
self._session.run()
File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
error = self.stream.connect()
File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
self.socket.do_handshake()
File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
return self._call_trampolining(
File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
return func(*a, **kw)
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
Versions:
focal:
ii python3-openvswitch 2.17.2-0ubuntu0.22.04.2~cloud0 all Python 3 bindings for Open vSwitch
ii python3-ovsdbapp 1.1.0-0ubuntu2 all library for creating OVSDB applications - Python 3.x
jammy:
ii python3-openvswitch 2.17.3-0ubuntu0.22.04.2 all Python 3 bindings for Open vSwitch
ii python3-ovsdbapp 1.15.1-0ubuntu2 all library for creating OVSDB applications - Python 3.x
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2018405/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list