[Bug 1657444] Re: Can't failover when rabbit_hosts is configured as 3 hosts

[Felipe Reyes]

On Thu, Jan 11, 2018 at 06:54:44PM -0000, Corey Bryant wrote:
> Felipe, would you be able to fill in the SRU details? [Impact], [Test
> Case], [Regression Potential] and then we'll need to subscribe the
> ubuntu-sru team.

I will, I was exactly writing those.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-oslo.messaging in Ubuntu.
https://bugs.launchpad.net/bugs/1657444

Title:
  Can't failover when rabbit_hosts is configured as 3 hosts

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive pike series:
  Triaged
Status in oslo.messaging:
  Fix Released
Status in python-oslo.messaging package in Ubuntu:
  Invalid
Status in python-oslo.messaging source package in Artful:
  Triaged

Bug description:
  I have a rabbitmq cluster of 3 nodes

  root at 47704165d2bb:/# rabbitmqctl cluster_status
  Cluster status of node rabbit at 47704165d2bb ...
  [{nodes,[{disc,[rabbit at 0482398a286e,rabbit at 3709521b608a,
                  rabbit at 47704165d2bb]}]},
   {running_nodes,[rabbit at 0482398a286e,rabbit at 3709521b608a,rabbit at 47704165d2bb]},
   {cluster_name,<<"rabbit at 47704165d2bb">>},
   {partitions,[]},
   {alarms,[{rabbit at 0482398a286e,[]},
            {rabbit at 3709521b608a,[]},
            {rabbit at 47704165d2bb,[]}]}]
  root at 47704165d2bb:/# rabbitmqctl list_policies      
  Listing policies ...
  /       ha-all  all     ^ha\\.  {"ha-mode":"all"}       0

  
  My oslo_message client configuration
  [oslo_messaging_rabbit]
  rabbit_hosts=120.0.0.56:5671,120.0.0.57:5671,120.0.0.55:5671
  rabbit_userid=cloud
  rabbit_password=cloud
  rabbit_ha_queues=True
  rabbit_retry_interval=1
  rabbit_retry_backoff=2
  rabbit_max_retries=0
  rabbit_durable_queues=False

  When I run "service rabbitmq-server stop" on one node to simulating a
  failure, I got following error logs, and the consumer can't failover
  from the bad node. It will reconnect the failure node forever instead
  of other nodes. "kombu_failover_strategy" is default value of "round-
  robin".


  2009-01-13 18:32:42.785 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [4e976d46-ceee-4617-b9be-5e4821990738] AMQP server 120.0.0.56:5671 closed the connection. Check login credentials: Socket closed
  2009-01-13 18:32:43.819 17 ERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on 120.0.0.56:5671 after None tries: Socket closed
  2009-01-13 18:32:43.819 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...
  2009-01-13 18:32:58.874 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [4e976d46-ceee-4617-b9be-5e4821990738] AMQP server 120.0.0.56:5671 closed the connection. Check login credentials: Socket closed
  2009-01-13 18:32:59.907 17 ERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on 120.0.0.56:5671 after None tries: Socket closed
  2009-01-13 18:32:59.907 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...

  
  Who can help me. Thanks!

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1657444/+subscriptions