[Bug 1985062] Re: ovsdbapp ssl send socket error

Alin-Gabriel Serdean 1985062 at bugs.launchpad.net
Tue Sep 27 23:40:49 UTC 2022 

Steps to reproduce

* Use a python client to connect to the Southbound Database.
* Once it is connected add an iptables rule to reject that traffic with TCP reset.
* Restart the application after installing the rule mentioned above and you will receive the following callstack:
File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 398, in run
    self._session.run()
  File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 532, in run
    self.__connect()
  File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 467, in __connect
    error, self.stream = ovs.stream.Stream.open(name)
  File "/usr/lib/python3/dist-packages/ovs/stream.py", line 196, in open
    err = cls.check_connection_completion(sock)
  File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
    return Stream.check_connection_completion(sock)
  File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
    return ovs.socket_util.check_connection_completion(sock)
  File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
    sock.send("\0".encode(), socket.MSG_DONTWAIT)
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
    return self._call_trampolining(
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
    return func(*a, **kw)
  File "/usr/lib/python3.8/ssl.py", line 1170, in send
    raise ValueError(
ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>


Example using Openstack + neutron-ovn-metadata-agent (ovsdbapp):
* Check a compute instance which has neutron-ovn-metadata-agent connected to the SB DB:
$ sudo netstat -tuapn | grep 6642
tcp        0      0 10.230.57.99:38834      10.230.62.255:6642      ESTABLISHED 2209992/ovn-control
tcp        0      0 10.230.57.99:59670      10.230.58.204:6642      ESTABLISHED 3687447/neutron-ovn
tcp        0      0 10.230.57.99:45296      10.230.58.185:6642      ESTABLISHED 3687446/neutron-ovn
tcp        0      0 10.230.57.99:59668      10.230.58.204:6642      ESTABLISHED 3687426/neutron-ovn
* Add iptable rule to reject traffic:
$ sudo iptables -A OUTPUT -p tcp --destination-port 6642 -j REJECT --reject-with tcp-reset
* Restart the neutron-ovn-metadata-agent:
$ sudo systemctl restart neutron-ovn-metadata-agent.service
* Check log at /var/log/neutron/neutron-ovn-metadata-agent.log

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1985062

Title:
  ovsdbapp ssl send socket error

Status in openvswitch:
  New
Status in openvswitch package in Ubuntu:
  New
Status in openvswitch source package in Jammy:
  New
Status in openvswitch source package in Kinetic:
  New

Bug description:
  ovsdbapps are sometimes failing with the following callstack:

  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection [req-d6abd5b4-69aa-4a93-ab42-82d80e75500e - - - - -] non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 817, in connect
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = super(SSLStream, self).connect()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 300, in connect
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.__scs_connecting()
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 268, in __scs_connecting
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = self.check_connection_completion(self.socket)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return Stream.check_connection_completion(sock)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return ovs.socket_util.check_connection_completion(sock)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     sock.send("\0".encode(), socket.MSG_DONTWAIT)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.8/ssl.py", line 1170, in send
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection     raise ValueError(
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.087 646221 ERROR ovsdbapp.backend.ovs_idl.connection 
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection [req-d6abd5b4-69aa-4a93-ab42-82d80e75500e - - - - -] non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 817, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = super(SSLStream, self).connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 300, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.__scs_connecting()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 268, in __scs_connecting
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = self.check_connection_completion(self.socket)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return Stream.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return ovs.socket_util.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     sock.send("\0".encode(), socket.MSG_DONTWAIT)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.8/ssl.py", line 1170, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     raise ValueError(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection 
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection During handling of the above exception, another exception occurred:
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection 
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 116, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     idlutils.wait_for_change(self.idl, self.timeout)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 51, in wait_for_change
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     while idl_.change_seqno == seqno and not idl_.run():
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 433, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/jsonrpc.py", line 519, in run
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 817, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = super(SSLStream, self).connect()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 300, in connect
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     self.__scs_connecting()
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 268, in __scs_connecting
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     retval = self.check_connection_completion(self.socket)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 777, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return Stream.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 137, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return ovs.socket_util.check_connection_completion(sock)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/ovs/socket_util.py", line 181, in check_connection_completion
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     sock.send("\0".encode(), socket.MSG_DONTWAIT)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 193, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return self._call_trampolining(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     return func(*a, **kw)
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.8/ssl.py", line 1170, in send
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection     raise ValueError(
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
  2022-08-10 00:00:22.088 646221 ERROR ovsdbapp.backend.ovs_idl.connection 
  2022-08-10 00:00:22.089 646221 INFO ovsdbapp.backend.ovs_idl.connection [req-d6abd5b4-69aa-4a93-ab42-82d80e75500e - - - - -] Trying to recover, sleeping 60 seconds

  The following relevant OVS packages are installed

  root at ps5-rb1-n1:~# dpkg -l | grep openvswitch
  ii  openvswitch-common                    2.17.2-0ubuntu0                                      amd64        Open vSwitch common components
  ii  openvswitch-switch                    2.17.2-0ubuntu0                                      amd64        Open vSwitch switch implementations
  ii  python3-openvswitch                   2.17.2-0ubuntu0                                      amd64        Python 3 bindings for Open vSwitch

  This should be fixed by the following patch proposed but not yet commited upstream:
  https://patchwork.ozlabs.org/project/openvswitch/patch/20220808173242.955238-1-mtomaska@redhat.com/

To manage notifications about this bug go to:
https://bugs.launchpad.net/openvswitch/+bug/1985062/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list