[Bug 1905423] Re: Rabbit reports "file descriptor limit alarm set", does not accept connections

[Sergio Durigan Junior]

Thank you for taking the time to file a bug report.

Would you be able to come up with a way for us to reproduce this bug?  I
understand that it must be happening in a production environment where
thousands of connections are being made, but maybe we can come up with a
way to reproduce it locally as well.

Does this bug happen often, or is it the first time it happened?

I tried finding some similar cases on the internet, but what I found
were reports of real scenarios where increasing the fd limit solved the
problem.  Curiously, one of the solutions was to add the LimitNOFILE
directive to the systemd unit file, which is already present in your
case.

I'm subscribing the Ubuntu Server team to this bug so that we can follow
it closely.

** Changed in: rabbitmq-server (Ubuntu)
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to rabbitmq-server in Ubuntu.
https://bugs.launchpad.net/bugs/1905423

Title:
  Rabbit reports "file descriptor limit alarm set", does not accept
  connections

Status in rabbitmq-server package in Ubuntu:
  Incomplete

Bug description:
  We had an incident where rabbitmq-server would stop accepting
  connections, with the following message in the log:

  =WARNING REPORT==== 24-Nov-2020::09:46:02 ===
  file descriptor limit alarm set.

  ********************************************************************
  *** New connections will not be accepted until this alarm clears ***
  ********************************************************************

  
  However, when checking the fd limits these are set quite high:

  root at juju-efcc41-12-lxd-0:~# cat /etc/default/rabbitmq-server
  # Generated by juju
  # bump ulimit so rabbit can support lots of connections
  ulimit -n 65536

  root at juju-efcc41-12-lxd-0:~# grep Limit /lib/systemd/system/rabbitmq-server.service
  LimitNOFILE=65536

  On the other hand, checking fd usage via lsof, `lsof -u rabbitmq`
  would only report abt. 900 open fds

  After bouncing rabbitmq it resumed accepting connects

  Versions:

  rabbitmq-server                  3.5.7-1ubuntu0.16.04.2 
  Ubuntu 16.04.5 LTS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1905423/+subscriptions