[Bug 1783203] Re: Upgrade to RabbitMQ 3.6.10 causes beam lockup in clustered deployment

[Andreas Hasenack]

The latest release in the 3.6.x series seems to be 3.6.16. Almost all
releases since 3.6.10 list changes to the management plugin, but I
didn't spot anything obvious. Here are the release notes for each:

https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_16
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_15
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_14
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_13
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_12
https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_11

Maybe this in 3.6.12? https://github.com/rabbitmq/rabbitmq-
server/issues/1346

Can anybody more familiar with the problem take a quick peek at the
release notes to see if something about this bug jumps out?


** Bug watch added: github.com/rabbitmq/rabbitmq-server/issues #1346
   https://github.com/rabbitmq/rabbitmq-server/issues/1346

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1783203

Title:
  Upgrade to RabbitMQ 3.6.10 causes beam lockup in clustered deployment

Status in OpenStack rabbitmq-server charm:
  New
Status in rabbitmq-server package in Ubuntu:
  Confirmed

Bug description:
  While performing an openstack release upgrade from Pike to Queens
  following the charmers guide, we have upgraded Ceph-* and Mysql.
  After setting source=cloud:xenial-queens on the RabbitMQ-Server charm
  and the cluster re-stabilizes, rabbitmq beam processes lock up on one
  cluster node causing complete denial of service on the openstack vhost
  across all 3 members of the cluster.  Killing the beam process on that
  node causes another node to lock up within a short timeframe.

  We have reproduced this twice in the same environment by re-deploying
  a fresh pike rabbitmq cluster and upgrading to queens.  The issue is
  not reproducable with generic workloads such as creating/deleting nova
  instances and creating/attaching/detaching cinder volumes, however,
  when running a full heat stack, we can reproduce this issue.

  This is happening on two of the three clouds on this site when RMQ is
  upgraded to Queens.  The third cloud was able to upgrade to Queens
  without issue but was upgraded on 18.02 charms.  Heat templates
  forthcoming.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1783203/+subscriptions