[Bug 1861457] Re: pyroute2 0.5.2 doesn't support neutron-common 14.0.4

Ryan Farrell 1861457 at bugs.launchpad.net
Fri Feb 7 14:54:50 UTC 2020 

Maybe it would help if we describe the flow of upgrades:

The cloud started with neutron packages at 14.0.2 on the computes, n-gw and n-api;
- A charm upgrade for trilio on the computes ran apt-upgrade, which moved the computes to neutron 14.0.4. This exposed a different know issue, which needed to be resolved by moving n-gw and n-api neutron packages to the same rev.

- We performed apt install neutron-common to n-api, and ngw, moving
neutron to 14.0.4 there as well.

- Still experiencing networking issues we ran apt-upgrade on n-gw and
n-api, and restarted all non cleanup neutron services. Then we observed
the log messages posted in this ticket, and finally resolved this issue
by forcing pyroute2 to 0.5.6, and removing and re-adding all preexisting
ha-routers from their l3-agent.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1861457

Title:
  pyroute2 0.5.2 doesn't support neutron-common 14.0.4

Status in OpenStack neutron-gateway charm:
  Invalid
Status in neutron package in Ubuntu:
  Incomplete

Bug description:
  On a Bionic-Stein clouds we recently updated all packages and
  encountered issues creating routers namespaces due to some changes in
  the data structure expected from pyroute2.

  For Bionic/Stein, the latest neutron-common package is from 14.0.4 and
  the latest pyroute2 is 0.5.2.

  We were able to resolve the issue by forcing an update of pyroute to 0.5.6 from the Eoan repository on n-gw units and cycling the neutron-openvswitch-agent and neutron-l3-agent services.
  Fixes will need to be back ported and released to Bionic Stein repo.

  
  2020-01-30 20:00:49.261 1874349 DEBUG oslo_concurrency.lockutils [-] Releasing lock "iptables-qrouter-5241d017-9d8a-44b4-912f-d36703bbc259" lock /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:281
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info [-] 255: KeyError: 255
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 158, in call
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 1188, in process
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     self.process_external()
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 946, in process_external
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     self.external_gateway_added(ex_gw_port, interface_name)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 434, in external_gateway_added
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     self._plug_external_gateway(ex_gw_port, interface_name, self.ns_name)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     mtu=ex_gw_port.get('mtu'))
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 267, in plug
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     bridge, namespace, prefix, mtu)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 355, in plug_new
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     namespace2=namespace)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 213, in add_veth
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     name1, self.namespace, 'veth', peer=peer)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     return input_func(*args, **kwargs)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 241, in _wrap
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     return self.channel.remote_call(name, args, kwargs)
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info   File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 203, in remote_call
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info     raise exc_type(*result[2])
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info KeyError: 255
  2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 5241d017-9d8a-44b4-912f-d36703bbc259: KeyError: 255
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 567, in _process_router_if_compatible
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self._process_added_router(router)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 575, in _process_added_router
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     ri.process()
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 485, in process
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     super(HaRouter, self).process()
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 161, in call
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self.logger(e)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self.force_reraise()
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     raise value
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 158, in call
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 1188, in process
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self.process_external()
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 946, in process_external
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self.external_gateway_added(ex_gw_port, interface_name)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 434, in external_gateway_added
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     self._plug_external_gateway(ex_gw_port, interface_name, self.ns_name)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     mtu=ex_gw_port.get('mtu'))
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 267, in plug
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     bridge, namespace, prefix, mtu)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 355, in plug_new
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     namespace2=namespace)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 213, in add_veth
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     name1, self.namespace, 'veth', peer=peer)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     return input_func(*args, **kwargs)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 241, in _wrap
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs)
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent   File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 203, in remote_call
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
  2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent KeyError: 255

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-gateway/+bug/1861457/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list