[Bug 1985062] Re: ovsdbapp ssl send socket error

Nobuto Murata 1985062 at bugs.launchpad.net
Fri Oct 14 16:24:48 UTC 2022 

** Tags removed: verification-done-focal

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1985062

Title:
  ovsdbapp ssl send socket error

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive yoga series:
  Fix Committed
Status in openvswitch:
  New
Status in openvswitch package in Ubuntu:
  Fix Released
Status in openvswitch source package in Jammy:
  Fix Committed
Status in openvswitch source package in Kinetic:
  Fix Released

Bug description:
  [Impact]
  Python clients that use python3-openvswitch i.e. ovsdbapp will fail on socket error with the callstack above while trying to connect to their corresponding servers

  [Test Case]
  * deploy Openstack Yoga
  * check on a compute instance which has neutron-ovn-metadata-agent connected to the SB DB:
  $ sudo netstat -tuapn | grep 6642
  tcp 0 0 10.230.57.99:38834 10.230.62.255:6642 ESTABLISHED 2209992/ovn-control
  tcp 0 0 10.230.57.99:59670 10.230.58.204:6642 ESTABLISHED 3687447/neutron-ovn
  tcp 0 0 10.230.57.99:45296 10.230.58.185:6642 ESTABLISHED 3687446/neutron-ovn
  tcp 0 0 10.230.57.99:59668 10.230.58.204:6642 ESTABLISHED 3687426/neutron-ovn
  * Add iptable rule to reject traffic for SB DB port 6442:
  $ sudo iptables -A OUTPUT -p tcp --destination-port 6642 -j REJECT --reject-with tcp-reset
  * Restart the neutron-ovn-metadata-agent:
  $ sudo systemctl restart neutron-ovn-metadata-agent.service
  * Check log at /var/log/neutron/neutron-ovn-metadata-agent.log

  [Where things could go wrong]
  Regression is not expected since this will just remove the unneeded socket options. The fix has also been applied upstream https://github.com/openvswitch/ovs/commit/fe27e0c8843ce37cd0f63ce19c11850e6865c85e however a new version has not been released.

  [Original bug description]

  ovsdbapps are sometimes failing with the following callstack:

  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection [req-d6abd5b4-69aa-4a93-ab42-82d80e75500e - - - - -] non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 817, in connect
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = super(SSLStream, self).connect()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 300, in connect
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.__scs_connecting()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 268, in __scs_connecting
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = self.check_connection_completion(self.socket)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return Stream.check_connection_completion(sock)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return ovs.socket_util.check_connection_completion(sock)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     sock.send("\0".encode(), socket.MSG_DONTWAIT)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.8/ssl.py", line 1170, in send
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     raise ValueError(
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection [req-d6abd5b4-69aa-4a93-ab42-82d80e75500e - - - - -] non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 817, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = super(SSLStream, self).connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 300, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.__scs_connecting()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 268, in __scs_connecting
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = self.check_connection_completion(self.socket)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return Stream.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return ovs.socket_util.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     sock.send("\0".encode(), socket.MSG_DONTWAIT)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.8/ssl.py", line 1170, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     raise ValueError(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection During handling of the above exception, another exception occurred:
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 116, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     idlutils.wait_for_change(self.idl, self.timeout)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 51, in wait_for_change
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     while idl_.change_seqno == seqno and not idl_.run():
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 817, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = super(SSLStream, self).connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 300, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.__scs_connecting()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 268, in __scs_connecting
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = self.check_connection_completion(self.socket)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return Stream.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return ovs.socket_util.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     sock.send("\0".encode(), socket.MSG_DONTWAIT)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.8/ssl.py", line 1170, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     raise ValueError(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection
  2022-08-10 00:00:22.089 646221 INFO ovsdbapp.backend.ovs_idl.connection [req-d6abd5b4-69aa-4a93-ab42-82d80e75500e - - - - -] Trying to recover, sleeping 60 seconds

  The following relevant OVS packages are installed

  root at ps5-rb1-n1:~# dpkg -l | grep openvswitch
  ii  openvswitch-common                    2.17.2-0ubuntu0                                      amd64        Open vSwitch common components
  ii  openvswitch-switch                    2.17.2-0ubuntu0                                      amd64        Open vSwitch switch implementations
  ii  python3-openvswitch                   2.17.2-0ubuntu0                                      amd64        Python 3 bindings for Open vSwitch

  This should be fixed by the following patch proposed but not yet commited upstream:
  https://patchwork.ozlabs.org/project/openvswitch/patch/20220808173242.955238-1-mtomaska@redhat.com/

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1985062/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list