[Bug 2018405] [NEW] neutron ovsdbapp ssl connection stuck in OSError error loop

Edward Hope-Morley 2018405 at bugs.launchpad.net
Wed May 3 14:43:24 UTC 2023 

Public bug reported:

We are running Openstack Yoga on both Ubuntu Focal and Ubuntu Jammy and
both environments are experiencing the following errors in neutron-
server and neutron-ovn-metadata-agent:

2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection [req-302f6be6-5b72-49b4-9132-d9ec39370fd5 - - - - -] [Errno 107] Transport endpoint is not connected: OSError: [Errno 107] Transport endpoint is not connected
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 108, in run
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.socket.do_handshake()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1337, in do_handshake
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self._check_connected()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1119, in _check_connected
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.getpeername()
2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection OSError: [Errno 107] Transport endpoint is not connected

This reconnect error happens forever after an initial ssl disconnect
like:

2023-04-12 19:15:40.281 2146189 ERROR ovsdbapp.backend.ovs_idl.transaction [req-7d895085-c4e1-43b9-a36f-808e86ef5caa f3797dd627d24377a7d0a1330595aed9 e4b04ca58d734ed0aa29e306adad4f79 - 93f9c955065f4cb38b9a8f1b98eedb92 93f9c955065f4cb38b9a8f1b98eedb92] Traceback (most recent
 call last):
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 114, in do_commit
    self.api.idl.run()
  File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
    self._session.run()
  File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
    error = self.stream.connect()
  File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
    self.socket.do_handshake()
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
    return self._call_trampolining(
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
    return func(*a, **kw)
  File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

Versions:
  focal:
    ii  python3-openvswitch                   2.17.2-0ubuntu0.22.04.2~cloud0                       all          Python 3 bindings for Open vSwitch
    ii  python3-ovsdbapp                      1.1.0-0ubuntu2                                       all          library for creating OVSDB applications - Python 3.x
  jammy:
    ii  python3-openvswitch             2.17.3-0ubuntu0.22.04.2                         all          Python 3 bindings for Open vSwitch
    ii  python3-ovsdbapp                1.15.1-0ubuntu2                                 all          library for creating OVSDB applications - Python 3.x

** Affects: openvswitch (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/2018405

Title:
  neutron ovsdbapp ssl connection stuck in OSError error loop

Status in openvswitch package in Ubuntu:
  New

Bug description:
  We are running Openstack Yoga on both Ubuntu Focal and Ubuntu Jammy
  and both environments are experiencing the following errors in
  neutron-server and neutron-ovn-metadata-agent:

  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection [req-302f6be6-5b72-49b4-9132-d9ec39370fd5 - - - - -] [Errno 107] Transport endpoint is not connected: OSError: [Errno 107] Transport endpoint is not connected
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 108, in run
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.socket.do_handshake()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1337, in do_handshake
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self._check_connected()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1119, in _check_connected
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection     self.getpeername()
  2023-05-02 00:01:05.026 2146189 ERROR ovsdbapp.backend.ovs_idl.connection OSError: [Errno 107] Transport endpoint is not connected

  This reconnect error happens forever after an initial ssl disconnect
  like:

  2023-04-12 19:15:40.281 2146189 ERROR ovsdbapp.backend.ovs_idl.transaction [req-7d895085-c4e1-43b9-a36f-808e86ef5caa f3797dd627d24377a7d0a1330595aed9 e4b04ca58d734ed0aa29e306adad4f79 - 93f9c955065f4cb38b9a8f1b98eedb92 93f9c955065f4cb38b9a8f1b98eedb92] Traceback (most recent
   call last):
    File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 114, in do_commit
      self.api.idl.run()
    File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
      self._session.run()
    File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
      error = self.stream.connect()
    File "/usr/lib/python3/dist-packages/ovs/stream.py", line 824, in connect
      self.socket.do_handshake()
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
      return self._call_trampolining(
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
      return func(*a, **kw)
    File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
      self._sslobj.do_handshake()
  ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

  Versions:
    focal:
      ii  python3-openvswitch                   2.17.2-0ubuntu0.22.04.2~cloud0                       all          Python 3 bindings for Open vSwitch
      ii  python3-ovsdbapp                      1.1.0-0ubuntu2                                       all          library for creating OVSDB applications - Python 3.x
    jammy:
      ii  python3-openvswitch             2.17.3-0ubuntu0.22.04.2                         all          Python 3 bindings for Open vSwitch
      ii  python3-ovsdbapp                1.15.1-0ubuntu2                                 all          library for creating OVSDB applications - Python 3.x

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2018405/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list