[Bug 1750121] Re: Dynamic routing: adding speaker to agent fails

David Ames david.ames at canonical.com
Mon May 7 17:55:44 UTC 2018 

I still see this failure when the neutron-bgp-dragent is restarted.

Bionic proposed: 12.0.0-0ubuntu1.1

The initial setup works fine:
2018-05-07 17:15:59.099 17215 INFO bgpspeaker.api.base [req-9692824c-b285-4304-86d2-00f46df8a216 - - - - -] API method core.start called with args: {'router_id': '10.5.0.82', 'label_range': (100, 100000), 'waiter'
: <ryu.lib.hub.Event object at 0x7f7cba5529d0>, 'bgp_server_port': 0, 'local_as': 12345, 'allow_local_as_in_count': 0, 'refresh_stalepath_time': 0, 'cluster_id': None, 'local_pref': 100, 'refresh_max_eor_time': 0}
2018-05-07 17:15:59.199 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-9692824c-b285-4304-86d2-00f46df8a216 - - - - -] Added BGP Speaker for local_as=12345 with router_id= 10.5.0.82. 
2018-05-07 17:16:00.689 17215 INFO bgpspeaker.api.base [req-6905ebee-595e-4a72-b2ac-d0372116f310 - - - - -] API method network.add called with args: {'prefix': u'192.168.0.0/24', 'next_hop': u'10.5.150.0'}        
2018-05-07 17:16:00.691 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-6905ebee-595e-4a72-b2ac-d0372116f310 - - - - -] Route cidr=192.168.0.0/24, nexthop=10.5.150.0 is advertised for 
BGP Speaker running for local_as=12345.                                                                                                                                                                              
2018-05-07 17:16:02.152 17215 INFO bgpspeaker.api.base [req-0f7a3721-db7e-418d-8ee7-0b950b4ddc88 103842446d8b4a029c1892ffb576d57d 15205edcf62643d7a3723ff7e23b74fc - - -] API method neighbor.create called with args
: {'connect_mode': 'active', 'cap_mbgp_evpn': False, 'remote_as': 10000, 'cap_mbgp_vpnv6': False, 'cap_mbgp_l2vpnfs': False, 'cap_four_octet_as_number': True, 'cap_mbgp_ipv6': False, 'is_next_hop_self': False, 'cap_mbgp_ipv4': True, 'cap_mbgp_ipv4fs': False, 'is_route_reflector_client': False, 'cap_mbgp_ipv6fs': False, 'is_route_server_client': False, 'cap_enhanced_refresh': False, 'peer_next_hop': None, 'password': None,
'ip_address': u'10.5.0.79', 'cap_mbgp_vpnv4fs': False, 'cap_mbgp_vpnv4': False, 'cap_mbgp_vpnv6fs': False}
2018-05-07 17:16:02.153 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-0f7a3721-db7e-418d-8ee7-0b950b4ddc88 103842446d8b4a029c1892ffb576d57d 15205edcf62643d7a3723ff7e23b74fc - - -] A$ded BGP Peer 10.5.0.79 for remote_as=10000 to BGP Speaker running for local_as=12345.
2018-05-07 17:16:03.158 17215 INFO bgpspeaker.peer [-] Connection to peer: 10.5.0.79 established
2018-05-07 17:16:03.159 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [-] BGP Peer 10.5.0.79 for remote_as=10000 is UP.
2018-05-07 17:16:04.167 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [-] Best path change observed. cidr=10.5.0.0/16, nexthop=10.5.0.79, remote_as=10000, is_withdraw=False
2018-05-07 17:16:04.169 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [-] Best path change observed. cidr=252.0.0.0/8, nexthop=10.5.0.79, remote_as=10000, is_withdraw=False
2018-05-07 17:16:08.756 17215 INFO bgpspeaker.api.base [req-55216404-ace9-4f46-8915-c952549a61db - - - - -] API method network.add called with args: {'prefix': u'10.5.150.9/32', 'next_hop': u'10.5.150.0'}
2018-05-07 17:16:08.761 17215 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-55216404-ace9-4f46-8915-c952549a61db - - - - -] Route cidr=10.5.150.9/32, nexthop=10.5.150.0 is advertised for $GP Speaker running for local_as=12345.

At this point the peer (quagga) has the expected routes via BGP.

The neutron-bgp-dragent is restarted:

2018-05-07 17:20:09.208 17885 INFO neutron.common.config [-] Logging enabled!
2018-05-07 17:20:09.209 17885 INFO neutron.common.config [-] /usr/bin/neutron-bgp-dragent version 12.0.1
2018-05-07 17:20:09.918 17885 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [-] Initializing Ryu driver for BGP Speaker functionality.
2018-05-07 17:20:09.918 17885 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [-] Initialized Ryu BGP Speaker driver interface with bgp_router_id=10.5.0.82
2018-05-07 17:20:09.926 17885 WARNING oslo_config.cfg [req-47f966f9-848c-4f7d-89dd-ff2cc60c001c - - - - -] Option "rabbit_host" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2018-05-07 17:20:09.927 17885 WARNING oslo_config.cfg [req-47f966f9-848c-4f7d-89dd-ff2cc60c001c - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2018-05-07 17:20:09.928 17885 WARNING oslo_config.cfg [req-47f966f9-848c-4f7d-89dd-ff2cc60c001c - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2018-05-07 17:20:09.928 17885 WARNING oslo_config.cfg [req-47f966f9-848c-4f7d-89dd-ff2cc60c001c - - - - -] Option "rabbit_virtual_host" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2018-05-07 17:20:10.015 17885 INFO neutron_dynamic_routing.services.bgp.agent.bgp_dragent [-] BGP dynamic routing agent started
2018-05-07 17:20:10.170 17885 INFO bgpspeaker.api.base [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] API method core.start called with args: {'router_id': '10.5.0.82', 'label_range': (100, 100000), 'waiter': <ryu.lib.hub.Event object at 0x7f48af8dd110>, 'bgp_server_port': 0, 'local_as': 12345, 'allow_local_as_in_count': 0, 'refresh_stalepath_time': 0, 'cluster_id': None, 'local_pref': 100, 'refresh_max_eor_time': 0}
2018-05-07 17:20:10.275 17885 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] Added BGP Speaker for local_as=12345 with router_id= 10.5.0.82.
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] 'auth_type': KeyError: 'auth_type'
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent Traceback (most recent call last):
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 161, in call
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     return func(*args, **kwargs)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 335, in safe_configure_dragent_for_bgp_speaker
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     self.add_bgp_speaker_on_dragent(bgp_speaker)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     result = f(*args, **kwargs)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 359, in add_bgp_speaker_on_dragent
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     self.add_bgp_peers_to_bgp_speaker(bgp_speaker)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     result = f(*args, **kwargs)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 390, in add_bgp_peers_to_bgp_speaker
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     bgp_peer)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     result = f(*args, **kwargs)
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent   File "/usr/lib/python2.7/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 408, in add_bgp_peer_to_bgp_speaker
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent     'auth_type': bgp_peer['auth_type']})
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent KeyError: 'auth_type'
2018-05-07 17:20:10.285 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent
2018-05-07 17:20:10.288 17885 ERROR neutron_dynamic_routing.services.bgp.agent.bgp_dragent [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] Unable to sync BGP speaker state.
2018-05-07 17:20:10.376 17885 INFO bgpspeaker.api.base [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] API method network.add called with args: {'prefix': u'10.5.150.9/32', 'next_hop': u'10.5.150.0'}
2018-05-07 17:20:10.380 17885 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] Route cidr=10.5.150.9/32, nexthop=10.5.150.0 is advertised for BGP Speaker running for local_as=12345.
2018-05-07 17:20:10.381 17885 INFO bgpspeaker.api.base [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] API method network.add called with args: {'prefix': u'192.168.0.0/24', 'next_hop': u'10.5.150.0'}
2018-05-07 17:20:10.382 17885 INFO neutron_dynamic_routing.services.bgp.agent.driver.ryu.driver [req-06883b21-9557-4706-8bc4-6d08b391a972 - - - - -] Route cidr=192.168.0.0/24, nexthop=10.5.150.0 is advertised for
BGP Speaker running for local_as=12345.


The peer relationship is dead. Removing the peer via openstack client and re-adding fixes the state of the BGP peer relationship.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1750121

Title:
  Dynamic routing: adding speaker to agent fails

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive pike series:
  Fix Committed
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in neutron:
  Fix Released
Status in neutron-dynamic-routing package in Ubuntu:
  Fix Released
Status in neutron-dynamic-routing source package in Artful:
  Fix Committed
Status in neutron-dynamic-routing source package in Bionic:
  Fix Committed
Status in neutron-dynamic-routing source package in Cosmic:
  Fix Released

Bug description:
  SRU details for Ubuntu
  ----------------------
  [Impact]
  See "Original description" below.

  [Test Case]
  See "Original description" below.

  [Regression Potential]
  Low. This is fixed upstream in corresponding stable branches.

  
  Original description
  --------------------
  When following https://docs.openstack.org/neutron-dynamic-routing/latest/contributor/testing.html everything works fine because the speaker is scheduled to the agent automatically (in contrast to what the docs say). But if I remove the speaker from the agent and add it again with

  $ neutron bgp-dragent-speaker-remove 0159fc0a-22de-4995-8fad-8fb8835a4d86 bgp-speaker
  $ neutron bgp-dragent-speaker-add 0159fc0a-22de-4995-8fad-8fb8835a4d86 bgp-speaker

  the following error is seen in the log:

  Feb 17 10:56:30 test-node01 neutron-bgp-dragent[18999]: ERROR
  neutron_dynamic_routing.services.bgp.agent.bgp_dragent [None req-
  da9a22ae-52a2-4be7-a3e8-2dc2dc970fdd admin admin] Call to driver for
  BGP Speaker d2aa5935-30c2-4369-83ee-b3a0ff77cc49 add_bgp_peer has
  failed with exception 'auth_type'.

  The same thing happens when there are multiple agents and one tries to
  add the speaker to one of the other agents.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1750121/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list