[Bug 1783203] Re: Upgrade to RabbitMQ 3.6.10 causes beam lockup in clustered deployment

[Dr. Jens Harbott]

>From the commit message https://github.com/rabbitmq/rabbitmq-
server/pull/1431 "Avoid infinite loop when dropping entries in the GM"
in 3.6.15 sounds interesting, as we do seem to see some kind of infinite
loop here. I don't have an explanation though as to why GM would be
interacting with the management plugin in the way we see.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1783203

Title:
  Upgrade to RabbitMQ 3.6.10 causes beam lockup in clustered deployment

Status in OpenStack rabbitmq-server charm:
  New
Status in rabbitmq-server package in Ubuntu:
  Confirmed

Bug description:
  While performing an openstack release upgrade from Pike to Queens
  following the charmers guide, we have upgraded Ceph-* and Mysql.
  After setting source=cloud:xenial-queens on the RabbitMQ-Server charm
  and the cluster re-stabilizes, rabbitmq beam processes lock up on one
  cluster node causing complete denial of service on the openstack vhost
  across all 3 members of the cluster.  Killing the beam process on that
  node causes another node to lock up within a short timeframe.

  We have reproduced this twice in the same environment by re-deploying
  a fresh pike rabbitmq cluster and upgrading to queens.  The issue is
  not reproducable with generic workloads such as creating/deleting nova
  instances and creating/attaching/detaching cinder volumes, however,
  when running a full heat stack, we can reproduce this issue.

  This is happening on two of the three clouds on this site when RMQ is
  upgraded to Queens.  The third cloud was able to upgrade to Queens
  without issue but was upgraded on 18.02 charms.  Heat templates
  forthcoming.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1783203/+subscriptions