[Bug 1905423] [NEW] Rabbit reports "file descriptor limit alarm set", does not accept connections

[Peter Sabaini]

Public bug reported:

We had an incident where rabbitmq-server would stop accepting
connections, with the following message in the log:

=WARNING REPORT==== 24-Nov-2020::09:46:02 ===
file descriptor limit alarm set.

********************************************************************
*** New connections will not be accepted until this alarm clears ***
********************************************************************


However, when checking the fd limits these are set quite high:

root at juju-efcc41-12-lxd-0:~# cat /etc/default/rabbitmq-server
# Generated by juju
# bump ulimit so rabbit can support lots of connections
ulimit -n 65536

root at juju-efcc41-12-lxd-0:~# grep Limit /lib/systemd/system/rabbitmq-server.service
LimitNOFILE=65536

On the other hand, checking fd usage via lsof, `lsof -u rabbitmq` would
only report abt. 900 open fds

After bouncing rabbitmq it resumed accepting connects

Versions:

rabbitmq-server                  3.5.7-1ubuntu0.16.04.2 
Ubuntu 16.04.5 LTS

** Affects: rabbitmq-server (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1905423

Title:
  Rabbit reports "file descriptor limit alarm set", does not accept
  connections

Status in rabbitmq-server package in Ubuntu:
  New

Bug description:
  We had an incident where rabbitmq-server would stop accepting
  connections, with the following message in the log:

  =WARNING REPORT==== 24-Nov-2020::09:46:02 ===
  file descriptor limit alarm set.

  ********************************************************************
  *** New connections will not be accepted until this alarm clears ***
  ********************************************************************

  
  However, when checking the fd limits these are set quite high:

  root at juju-efcc41-12-lxd-0:~# cat /etc/default/rabbitmq-server
  # Generated by juju
  # bump ulimit so rabbit can support lots of connections
  ulimit -n 65536

  root at juju-efcc41-12-lxd-0:~# grep Limit /lib/systemd/system/rabbitmq-server.service
  LimitNOFILE=65536

  On the other hand, checking fd usage via lsof, `lsof -u rabbitmq`
  would only report abt. 900 open fds

  After bouncing rabbitmq it resumed accepting connects

  Versions:

  rabbitmq-server                  3.5.7-1ubuntu0.16.04.2 
  Ubuntu 16.04.5 LTS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1905423/+subscriptions