[Bug 1157916] Re: openstack hangs on message passing

[James Page]

Magosányi

Apologies for the lack for response on this bug; I note that this may be
from a much older OpenStack release (quantum -> neutron in newer
released).

I'm assuming that this is probably resolved (we use neutron all of the
time and I've never seen this problem), but please set back to NEW if
you think this is still an issue in OpenStack >= icehouse.

** Changed in: cloud-archive
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ubuntu-cloud-archive.
https://bugs.launchpad.net/bugs/1157916

Title:
  openstack hangs on message passing

Status in ubuntu-cloud-archive:
  Fix Released

Bug description:
  I can reproduce it both with quantum ('quantum net-create --tenant_id
  31fd997c87c04cae9cf15e4f2f305def demo-net --provider:network_type gre
  --provider:segmentation_id 1') and nova ('nova boot --image 617ce1fd-
  884d-49cf-94c6-083dfad6ac95 test').

  When I run the command, it hangs forever. In one instance I left it
  for hours.

  The underlying problem lies somewhere around quantum/openstack/common/rpc/amqp.py in notify(),
  maybe in the underlying kombu or amqplib.
  The teardown method of the ConnectionContext waits for some answer on the connection which never arrives.

  A stack trace to right before some read that blocks (off with a couple of lines because debug info inserted):
    File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
      func(*args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 567, in process_request
      proto = self.protocol(socket, address, self)

    File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
      self.handle()

    File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
      self.handle_one_request()

    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 278, in handle_one_request
      self.handle_one_response()

    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 382, in handle_one_response
      result = self.application(self.environ, start_response)

    File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
      return app(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
      return self.app(env, start_response)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
      return resp(environ, start_response)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
      resp = self.call_func(req, *args, **self.kwargs)

    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
      return self.func(req, *args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/quantum/api/v2/resource.py", line 97, in resource
      result = method(request=request, **args)

    File "/usr/lib/python2.7/dist-packages/quantum/api/v2/base.py", line 255, in create
      body)

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/notifier/api.py", line 139, in notify
      driver.notify(context, msg)

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/notifier/rabbit_notifier.py", line 43, in notify
      rpc.notify(context, topic, message)

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/__init__.py", line 190, in notify
      return _get_impl().notify(cfg.CONF, context, topic, msg)

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 764, in notify
      rpc_amqp.get_connection_pool(conf, Connection))

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 424, in notify
      LOG.debug("notify sent")

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 132, in __exit__
      self._done()

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 117, in _done
      self.connection.reset()

    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 553, in reset
      self.channel.close()

    File "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 229, in close
      super(Channel, self).close()

    File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 203, in close
      (20, 41),    # Channel.close_ok

    File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
      self.channel_id, allowed_methods)

    File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 206, in _wait_method
      AMQP_LOGGER.debug("before read. stack: %s"%(traceback.format_stack(),))

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1157916/+subscriptions