[Bug 1834565] Re: Services not running that should be: cinder-scheduler, neutron-lbaasv2-agent, neutron-dhcp-agent, neutron-metering-agent

Brian Murray brian at ubuntu.com
Tue Jul 2 20:45:44 UTC 2019 

Hello Chris, or anyone else affected,

Accepted python-eventlet into disco-proposed. The package will build now
and be available at https://launchpad.net/ubuntu/+source/python-
eventlet/0.24.1-0ubuntu3.1 in a few hours, and then in the -proposed
repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-disco to verification-done-disco. If it does not fix
the bug for you, please add a comment stating that, and change the tag
to verification-failed-disco. In either case, without details of your
testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: python-eventlet (Ubuntu Disco)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-disco

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-eventlet in Ubuntu.
https://bugs.launchpad.net/bugs/1834565

Title:
  Services not running that should be: cinder-scheduler, neutron-
  lbaasv2-agent, neutron-dhcp-agent, neutron-metering-agent

Status in OpenStack cinder charm:
  Invalid
Status in OpenStack neutron-gateway charm:
  Invalid
Status in Ubuntu Cloud Archive:
  In Progress
Status in Ubuntu Cloud Archive stein series:
  In Progress
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in python-eventlet package in Ubuntu:
  Fix Released
Status in python-eventlet source package in Disco:
  Fix Committed
Status in python-eventlet source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Unable to enable TLS termination of any endpoints for OpenStack with Python 3.7.

  [Test Case]
  sudo apt install python3-eventlet
  python3 << EOF
  import eventlet
  eventlet.monkey_patch()

  import socket
  import ssl

  sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  wrappedSocket = ssl.wrap_socket(sock)
  EOF

  Traceback (most recent call last):
    File "<stdin>", line 8, in <module>
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 403, in wrap_socket
      return GreenSSLSocket(sock, *a, **kw)
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
      *args, **kw
    File "/usr/lib/python3.7/ssl.py", line 1222, in wrap_socket
      suppress_ragged_eofs=suppress_ragged_eofs
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 426, in wrap_socket
      return GreenSSLSocket(sock, *a, _context=self, **kw)
    File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
      *args, **kw
  TypeError: wrap_socket() got an unexpected keyword argument '_context'

  [Regression Potential]
  Low; codepaths are all Python 3.7 specific due to internal changes in the SSL implementation.

  [Original bug report]
  In an SSL enabled OpenStack Stein deploy, deployed on Disco, the cinder-scheduler service is crashing with the below stacktrace:

  2019-06-27 18:35:07.593 5427 WARNING cinder.scheduler.host_manager [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] volume service is down. (host: juju-20eff5-mojo-0-7 at LVM)
  2019-06-27 18:35:07.594 5427 WARNING cinder.scheduler.host_manager [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] volume service is down. (host: cinder at cinder-ceph)
  2019-06-27 18:35:07.610 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-volume objects version 1.37 as minimum service version.
  2019-06-27 18:35:07.619 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-volume RPC version 3.16 as minimum service version.
  2019-06-27 18:35:07.638 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-scheduler objects version 1.37 as minimum service version.
  2019-06-27 18:35:07.647 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-scheduler RPC version 3.11 as minimum service version.
  2019-06-27 18:35:07.702 5427 WARNING oslo_config.cfg [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S  release.).  Its value may be silently ignored in the future.
  2019-06-27 18:35:07.703 5427 WARNING oslo_config.cfg [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Deprecated: Option "signing_dir" from group "keystone_authtoken" is deprecated for removal (PKI token format is no longer supported.).  Its value may be silently ignored in the future.
  2019-06-27 18:35:07.705 5427 WARNING oslo_config.cfg [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken".
  2019-06-27 18:35:07.726 5427 INFO cinder.service [-] Starting cinder-scheduler node (version 14.0.0)
  2019-06-27 18:35:07.736 5427 INFO cinder.manager [req-e208c3e5-83e5-4c5b-85a3-1f5e09c86da9 - - - - -] Initiating service 4 cleanup
  2019-06-27 18:35:07.745 5427 INFO cinder.manager [req-e208c3e5-83e5-4c5b-85a3-1f5e09c86da9 - - - - -] Service 4 cleanup completed.
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service [req-e208c3e5-83e5-4c5b-85a3-1f5e09c86da9 - - - - -] Error starting thread.: TypeError: wrap_socket() got an unexpected keyword argument '_context'
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service Traceback (most recent call last):
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     service.start()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/cinder/service.py", line 236, in start
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.rpcserver.start()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 270, in wrapper
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     log_after, timeout_timer)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 190, in run_once
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     post_fn = fn()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 269, in <lambda>
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     states[state].run_once(lambda: fn(self, *args, **kwargs),
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 414, in start
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.listener = self._create_listener()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 151, in _create_listener
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return self.transport._listen(self._target, 1, None)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 143, in _listen
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     batch_timeout)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 652, in listen
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in _get_connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     purpose=purpose)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/common.py", line 407, in __init__
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.connection = connection_pool.create(purpose)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/pool.py", line 144, in create
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return self.connection_cls(self.conf, self.url, purpose)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 531, in __init__
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.ensure_connection()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 646, in ensure_connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.connection.ensure_connection(errback=on_error)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 405, in ensure_connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     callback)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/kombu/utils/functional.py", line 332, in retry_over_time
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return fun(*args, **kwargs)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 261, in connect
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return self.connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 802, in connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self._connection = self._establish_connection()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 757, in _establish_connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     conn = self.transport.establish_connection()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     conn.connect()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 307, in connect
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.transport.connect()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/amqp/transport.py", line 83, in connect
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.socket_settings, self.read_timeout, self.write_timeout,
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/amqp/transport.py", line 187, in _init_socket
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self._setup_transport()
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/amqp/transport.py", line 302, in _setup_transport
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     self.sock = self._wrap_socket(self.sock, **self.sslopts)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/amqp/transport.py", line 309, in _wrap_socket
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return self._wrap_socket_sni(sock, **sslopts)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/amqp/transport.py", line 344, in _wrap_socket_sni
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     sock = ssl.wrap_socket(**opts)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 403, in wrap_socket
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return GreenSSLSocket(sock, *a, **kw)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     *args, **kw
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3.7/ssl.py", line 1222, in wrap_socket
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     suppress_ragged_eofs=suppress_ragged_eofs
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 426, in wrap_socket
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     return GreenSSLSocket(sock, *a, _context=self, **kw)
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service     *args, **kw
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service TypeError: wrap_socket() got an unexpected keyword argument '_context'
  2019-06-27 18:35:07.776 5427 ERROR oslo_service.service

  Additionally, charm-neutron-gateway has the same stacktrace in the
  mentioned services.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-cinder/+bug/1834565/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list