[Bug 1772236] Re: rabbit died and everything else died

[Iain Lane]

For autopkgtest-cloud I just cowboyed a change to add Restart=on-failure
to rabbitmq-server.service. Maybe that helps us mitigate, in combination
with delivery_mode=2.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1772236

Title:
  rabbit died and everything else died

Status in Auto Package Testing:
  New
Status in rabbitmq-server package in Ubuntu:
  New

Bug description:
  Why did it die?

  Should it have self-restarted?

  ubuntu at juju-prod-ues-proposed-migration-machine-1:~$ journalctl -u rabbitmq-server.service -n1000 | cat
  -- Logs begin at Sun 2018-05-20 00:18:25 UTC, end at Sun 2018-05-20 08:58:27 UTC. --
  May 20 04:00:11 juju-prod-ues-proposed-migration-machine-1 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=137/n/a
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: Stopping and halting node 'rabbit at ps45-10-25-180-146' ...
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: Error: unable to connect to node 'rabbit at ps45-10-25-180-146': nodedown
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: DIAGNOSTICS
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: ===========
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: attempted to contact: ['rabbit at ps45-10-25-180-146']
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: rabbit at ps45-10-25-180-146:
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]:   * connected to epmd (port 4369) on ps45-10-25-180-146
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]:   * epmd reports: node 'rabbit' not running at all
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]:                   other nodes on ps45-10-25-180-146: ['rabbitmq-cli-28979']
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]:   * suggestion: start the node
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: current node details:
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: - node name: 'rabbitmq-cli-28979 at juju-prod-ues-proposed-migration-machine-1'
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: - home dir: .
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 rabbitmq[28971]: - cookie hash: 7+AChRZDewWFJK8SEUhx+Q==
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=2
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 systemd[1]: rabbitmq-server.service: Unit entered failed state.
  May 20 04:00:12 juju-prod-ues-proposed-migration-machine-1 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1772236/+subscriptions