[Bug 1784757] [rabbitmq-server/bionic] verification still needed

[Brian Murray]

The fix for this bug has been awaiting testing feedback in the -proposed
repository for bionic for more than 90 days.  Please test this fix and
update the bug appropriately with the results.  In the event that the
fix for this bug is still not verified 15 days from now, the package
will be removed from the -proposed repository.

** Tags added: removal-candidate

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1784757

Title:
  [SRU] rabbitmq-server does not properly shutdown

Status in rabbitmq-server package in Ubuntu:
  Fix Released
Status in rabbitmq-server source package in Bionic:
  Fix Committed
Status in rabbitmq-server source package in Cosmic:
  Won't Fix
Status in rabbitmq-server package in Debian:
  Fix Released

Bug description:
  [Impact]

  The systemd file rabbitmq-server.service on Bionic uses "Type=simple"
  when defining the service, but unfortunately this doesn't work very
  well for rabbitmq-server. In certain situations, systemd will fail to
  keep track of a start/stop/restart event, and will hang for 90 seconds
  before giving the prompt back to the user. Another problem is that
  rabbitmq-server must start after the epmd service, so we need to
  explicitly declare this dependency in the service file.

  [Test Case]

  Although I was able to reproduce this almost 100% of the time, there
  were rare occasions when the restart procedure finished normally. I
  was also only able to reproduce it using a bionic VM, not a container.
  If you have multipass or lxd configured to launch VMs, that should be
  easy.

  The steps are:

  $ lxc launch ubuntu-daily:bionic --vm bug1784757-rabbitmq-server # or use multipass
  $ lxc shell bug1784757-rabbitmq-server
  # apt update
  # apt install rabbitmq-server -y
  # systemctl restart rabbitmq-server.service

  In a normal scenario, the restart should take around 3 seconds or
  less. With the bug, it takes around 90 seconds. If you can't reproduce
  it, try running "systemctl restart" again. A quick way to trigger it
  is to run a for loop like:

  # for i in $(seq 10); do time systemctl restart rabbitmq-
  server.service ; done

  [Regression Potential]

  * Because rabbitmq-server implements systemd's "Type=notify" using
  socat to communicate with systemd-notify over a socket, we will be
  introducing another point of failure (socat) in the mix.

  * So, albeit unlikely, there may be a problem when using socat.  That
  would not be a regression, though, since the outcome would be the same
  as we have today: "systemctl restart" would not properly work, even
  though the service did restart.

  [Original Report]

  When I run `systemctl restart rabbitmq-server` it waits for 90 seconds
  then systemd sends SIGKILL to it.

  Presumably the `epmd` process does not receive SIGTERM, since if I run
  `kill 1493` (or whatever pid it currently is) then restart happens
  straight after that successfully

  ● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
    Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
    Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
    Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
   Main PID: 177 (code=killed, signal=TERM)
      Tasks: 1 (limit: 4915)
     CGroup: /system.slice/rabbitmq-server.service
             └─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon

  Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
  Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit at rmq-1'
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
  Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
  Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
  Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit at rmq-1'

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1784757/+subscriptions