[Bug 2018405] Re: [SRU] neutron ovsdbapp ssl connection stuck in OSError error loop

Thu Jun 15 20:42:29 UTC 2023

Moreover, I think you can try to improve your "Where problems could
occur"/"Regression potential" section in the bug description. For
instance, the patch is adding more exceptions to be handled, maybe
another problem (not the one initially considered) might raise the same
exception and it might be treated in an unexpected way. I am pretty sure
upstream thought about that before merging the changes but this section
is exactly to elaborate on all of that, and if a regression happens we
can at least have a roughly idea of what happened. We wouldn't release
an update if we think it could regress users, so this is more of an
exercise of what could hypothetically happen.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/2018405

Title:
  [SRU] neutron ovsdbapp ssl connection stuck in OSError error loop

Status in openvswitch package in Ubuntu:
  New
Status in openvswitch source package in Jammy:
  New
Status in openvswitch source package in Kinetic:
  New
Status in openvswitch source package in Lunar:
  New
Status in openvswitch source package in Mantic:
  New

Bug description:
  [Impact]
  terminated/closed ssl connections between openstack agents and the ovn-central databases are not properly handled and this patch fixes that.

  [Test Plan]
  * deploy Openstack with OVN
  * abruptly terminate connections between neutron-server and/or neutron-ovn-metadata-agent and ovn-central NB/SB database servers.
  * ensure that connections are re-established and the errors below no not appear.

  [Regression Potential]
  No regressions are anticipated as a result of using this patch.

  ---------------------------------------------------------------------

  We are running Openstack Yoga on both Ubuntu Focal and Ubuntu Jammy
  and both environments are experiencing the following errors in
  neutron-server and neutron-ovn-metadata-agent:

  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection [req-302f6be6-5b72-49b4-9132-d9ec39370fd5 - - - - -] [Errno 107] Transport endpoint is not connected: OSError: [Errno 107] Transport endpoint is not connected
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 108, in run
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.socket.do_handshake()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1337, in do_handshake
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self._check_connected()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1119, in _check_connected
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.getpeername()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection OSError: [Errno 107] Transport endpoint is not connected

  This reconnect error happens forever after an initial ssl disconnect
  like:

  2023-04-12 19:15:40.281 2146189 ERROR ovsdbapp.backend.ovs_idl.transaction [req-7d895085-c4e1-43b9-a36f-808e86ef5caa f3797dd627d24377a7d0a1330595aed9 e4b04ca58d734ed0aa29e306adad4f79 - 93f9c955065f4cb38b9a8f1b98eedb92 93f9c955065f4cb38b9a8f1b98eedb92] Traceback (most recent
   call last):
    File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 114, in do_commit
      self.api.idl.run()
    File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
      self._session.run()
    File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
      error = self.stream.connect()
    File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
      self.socket.do_handshake()
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
      return self._call_trampolining(
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
      return func(*a, **kw)
    File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
      self._sslobj.do_handshake()
  ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

  Versions:
    focal:
      ii  python3-openvswitch                   2.17.2-0ubuntu0.22.04.2~cloud0                       all          Python 3 bindings for Open vSwitch
      ii  python3-ovsdbapp                      1.1.0-0ubuntu2                                       all          library for creating OVSDB applications - Python 3.x
    jammy:
      ii  python3-openvswitch             2.17.3-0ubuntu0.22.04.2                         all          Python 3 bindings for Open vSwitch
      ii  python3-ovsdbapp                1.15.1-0ubuntu2                                 all          library for creating OVSDB applications - Python 3.x

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2018405/+subscriptions