[Bug 1573073] Re: When router has no ports _process_updated_router fails because the namespace does not exist

Mon Nov 7 12:37:59 UTC 2016

Reviewed:  https://review.openstack.org/393739
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5a3dadc60103a8c2dc88446fe4b2412c9bdf0790
Submitter: Jenkins
Branch:    stable/newton

commit 5a3dadc60103a8c2dc88446fe4b2412c9bdf0790
Author: Saverio Proto <saverio.proto at switch.ch>
Date:   Thu Apr 21 16:50:07 2016 +0200

    Check if namespace exists before getting devices

    If a neutron router has no ports defined, its namespace is deleted
    by the cronjob /etc/cron.d/neutron-l3-agent-netns-cleanup.
    It is required to check if the namespace exists before calling
    iproute2 commands against the namespace.
    It is not enough for the string of the namespace uuid to be defined,
    the namespace must really exist on the network node.
    This patch checks if the namespace exists when calling get_devices().
    Otherwise the agent log file will be flooded with messages like
    Cannot open network namespace "qrouter-<uuid>": No such file or directory
    Failed to process compatible router '<uuid>'

    Related-bug: 1573073

    Change-Id: I744ef11529f9da5cbfdb812de0b35b95f9d078bb
    (cherry picked from commit 7675495f260daf0174f4e04b36279513889df19a)

** Tags added: in-stable-newton

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1573073

Title:
  When router has no ports _process_updated_router fails because the
  namespace does not exist

Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive liberty series:
  Triaged
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive newton series:
  Triaged
Status in neutron:
  In Progress
Status in neutron package in Ubuntu:
  Triaged
Status in neutron source package in Xenial:
  Triaged
Status in neutron source package in Yakkety:
  Triaged
Status in neutron source package in Zesty:
  Triaged

Bug description:
  Happens in Kilo. Cannot test on other releases.

  Steps to reproduce:

  1) create a router and set at least a port, also the gateway is fine
  2) check that the namespace exists with
     ip netns show | grep qrouter-<uuid>
  3) check the ports are there
     ip netns exec qrouter-<uuid> ip addr show
  4) delete all ports from the router
  5) check that only loopback interface is present
     ip netns exec qrouter-<uuid> ip addr show
  6) run the cronjob task that is installed in the file
     /etc/cron.d/neutron-l3-agent-netns-cleanup
      so basically run this command:
     /usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini
  7) the namespace should be gone:
     ip netns show | grep qrouter-<uuid>
  8) delete the neutron router.
  9) check log file /var/log/neutron/vpn-agent.log

  When the router has no ports the namespace is deleted from the network node by the cronjob. However this brakes the router updates and the file vpn-agent.log is flooded with this traces:

  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     return func(*args, **kwargs)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     self._process_internal_ports()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     raise RuntimeError(m)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in _process_router_update
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in _process_router_if_compatible
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_updated_router(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in _process_updated_router
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ri.process(self)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self.logger(e)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_internal_ports()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     raise RuntimeError(m)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1573073/+subscriptions