[Bug 1784757] Re: rabbitmq-server does not properly shutdown

[Alex Tomkins]

Fixes in Debian packaging which also address the problem:

- https://salsa.debian.org/openstack-team/third-party/rabbitmq-server/commit/9764f7e6ffeaa99325b8e5ff41e80b1c4b2894d3
- https://salsa.debian.org/openstack-team/third-party/rabbitmq-server/commit/c5b99f5c4811f460b73c8c8505a98fa55dcd752b

If I start the server with just `service rabbitmq-server start`, my
process tree looks like:

rabbitmq 11316  0.0  0.1   4628   800 ?        Ss   14:59   0:00   /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 11337  0.0  0.3   4628  1792 ?        S    14:59   0:00     /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 11633  8.8 12.4 2165876 61504 ?       Sl   14:59   0:02       /usr/lib/erlang/erts-9.2/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/ebin -noshell -noinput -s rabbit boot -sname rabbit at golestandt -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit at golestandt.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit at golestandt-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit at golestandt-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at golestandt" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 11745  0.1  0.1   4520   728 ?        Ss   14:59   0:00         erl_child_setup 65536
rabbitmq 11809  0.0  0.2   8264  1072 ?        Ss   14:59   0:00           inet_gethost 4
rabbitmq 11810  0.0  0.3  10384  1660 ?        S    14:59   0:00             inet_gethost 4
rabbitmq 11504  0.0  0.0  26852   204 ?        S    14:59   0:00   /usr/lib/erlang/erts-9.2/bin/epmd -daemon

However, doing `service epmd start` before that gives:

epmd     12226  0.0  0.3  26852  1728 ?        Ss   15:01   0:00   /usr/bin/epmd -systemd
rabbitmq 12263  0.0  0.1   4628   816 ?        Ss   15:01   0:00   /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 12287  0.0  0.3   4628  1736 ?        S    15:01   0:00     /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 12582 25.5 13.2 2171584 65160 ?       Sl   15:01   0:02       /usr/lib/erlang/erts-9.2/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/ebin -noshell -noinput -s rabbit boot -sname rabbit at golestandt -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit at golestandt.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit at golestandt-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit at golestandt-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit at golestandt" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 12728  0.2  0.1   4520   760 ?        Ss   15:01   0:00         erl_child_setup 65536
rabbitmq 12757  0.0  0.2   8264  1156 ?        Ss   15:01   0:00           inet_gethost 4
rabbitmq 12758  0.0  0.3  10384  1620 ?        S    15:01   0:00             inet_gethost 4

Which allows RabbitMQ to shutdown cleanly.

The service file probably needs dependency fixes.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1784757

Title:
  rabbitmq-server does not properly shutdown

Status in rabbitmq-server package in Ubuntu:
  Incomplete

Bug description:
  When I run `systemctl restart rabbitmq-server` it waits for 90 seconds
  then systemd sends SIGKILL to it.

  Presumably the `epmd` process does not receive SIGTERM, since if I run
  `kill 1493` (or whatever pid it currently is) then restart happens
  straight after that successfully

  ● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
    Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
    Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
    Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
   Main PID: 177 (code=killed, signal=TERM)
      Tasks: 1 (limit: 4915)
     CGroup: /system.slice/rabbitmq-server.service
             └─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon

  Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
  Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit at rmq-1'
  Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
  Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
  Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
  Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit at rmq-1'

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1784757/+subscriptions