[Bug 1993149] Re: VMs stay stuck in scheduling when rabbitmq leader unit is down

[Corey Bryant]

I've added the package and upstream oslo.messaging projects which need
fixing. I've triaged the rabbitmq charm as low since it is affected but
doesn't require a fix.

** Also affects: python-oslo.messaging (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: python-oslo.messaging (Ubuntu Kinetic)
   Importance: Critical
     Assignee: Corey Bryant (corey.bryant)
       Status: Triaged

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/zed
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/yoga
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/yoga
       Status: New => Triaged

** Changed in: cloud-archive/zed
       Status: New => Triaged

** Changed in: cloud-archive/zed
   Importance: Undecided => Critical

** Changed in: cloud-archive/yoga
   Importance: Undecided => Critical

** Changed in: python-oslo.messaging (Ubuntu Jammy)
       Status: New => Triaged

** Changed in: python-oslo.messaging (Ubuntu Jammy)
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-oslo.messaging in Ubuntu.
https://bugs.launchpad.net/bugs/1993149

Title:
  VMs stay stuck in scheduling when rabbitmq leader unit is down

Status in OpenStack RabbitMQ Server Charm:
  Triaged
Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive yoga series:
  Triaged
Status in Ubuntu Cloud Archive zed series:
  Triaged
Status in oslo.messaging:
  New
Status in python-oslo.messaging package in Ubuntu:
  Triaged
Status in python-oslo.messaging source package in Jammy:
  Triaged
Status in python-oslo.messaging source package in Kinetic:
  Triaged

Bug description:
  When testing rabbitmq-server HA in our OpenStack Yoga cloud
  environment (Rabbitmq Server release 3.9/stable) we faced the
  following issues:

  - When the leader unit is down we are unable to launch any VMs and the
  launched ones stay stuck in the 'BUILD' state.

  - While checking the logs we see that several OpenStack services has
  issues in communicating with the rabbitmq-server

  - After restarting all the services using rabbitmq (like Nova, Cinder,
  Neutron etc) the issue gets resolved and the VMs can be launched
  successfully

  
  The corresponding logs are available at: https://pastebin.ubuntu.com/p/Bk3yktR8tp/

  
  We also observed the same for rabbitmq-server unit which is first in the list of 'nova.conf' file, and after restarting the concerned rabbitmq unit we see that scheduling of VMs work fine again.

  As this can be seen from this part of the log as well:
  "Reconnected to AMQP server on 192.168.34.251:5672 via [amqp] client with port 41922."

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1993149/+subscriptions