[Bug 1573073] Related fix merged to neutron (stable/newton)
OpenStack Infra
1573073 at bugs.launchpad.net
Thu Apr 20 18:47:55 UTC 2017
Reviewed: https://review.openstack.org/457468
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=44c3ee8cb2bc9adb6f4f95ba3f6be4c3abe344df
Submitter: Jenkins
Branch: stable/newton
commit 44c3ee8cb2bc9adb6f4f95ba3f6be4c3abe344df
Author: Saverio Proto <saverio.proto at switch.ch>
Date: Mon Mar 27 16:39:16 2017 +0200
Dont try to apply iptables rules in a endless loop
If the namespace does not exist the current behavior
is to try to apply the iptables rules forever in a
endless loop. This fills up the logs on the network
node and leads to outage.
Change-Id: I628b18a66f9478d7349fa1817431aae2f62ee105
Related-bug: #1623664
Related-bug: #1573073
(cherry picked from commit 3889b0f214fbb1d32531395cff232cf57daf7b17)
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1573073
Title:
[SRU] When router has no ports _process_updated_router fails because
the namespace does not exist
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive liberty series:
Fix Released
Status in Ubuntu Cloud Archive mitaka series:
Fix Released
Status in Ubuntu Cloud Archive newton series:
Fix Released
Status in neutron:
In Progress
Status in neutron package in Ubuntu:
Fix Released
Status in neutron source package in Xenial:
Fix Released
Status in neutron source package in Yakkety:
Fix Released
Status in neutron source package in Zesty:
Fix Released
Bug description:
[Description]
[Testcase]
Happens in Kilo. Cannot test on other releases.
Steps to reproduce:
1) create a router and set at least a port, also the gateway is fine
2) check that the namespace exists with
ip netns show | grep qrouter-<uuid>
3) check the ports are there
ip netns exec qrouter-<uuid> ip addr show
4) delete all ports from the router
5) check that only loopback interface is present
ip netns exec qrouter-<uuid> ip addr show
6) run the cronjob task that is installed in the file
/etc/cron.d/neutron-l3-agent-netns-cleanup
so basically run this command:
/usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini
7) the namespace should be gone:
ip netns show | grep qrouter-<uuid>
8) delete the neutron router.
9) check log file /var/log/neutron/vpn-agent.log
When the router has no ports the namespace is deleted from the network
node by the cronjob. However this brakes the router updates and the
file vpn-agent.log is flooded with this traces:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info self._process_internal_ports()
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info existing_devices = self._get_existing_devices()
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info raise RuntimeError(m)
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in _process_router_update
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in _process_router_if_compatible
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent self._process_updated_router(router)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in _process_updated_router
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent ri.process(self)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent self.logger(e)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent self._process_internal_ports()
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent existing_devices = self._get_existing_devices()
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent raise RuntimeError(m)
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
[Regression Potential]
The patches that we're picking up in our packages have already landed upstream in master (zesty), stable/newton (yakkety), and stable/mitaka (xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1573073/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list