[Bug 1379739] [NEW] Neutron daemons unable to manage network namespaces

James Page james.page at ubuntu.com
Fri Oct 10 10:24:47 UTC 2014 

Public bug reported:

I saw this on upgrading to 2014.1.3; however downgrading was not
effective; stopping all daemons, manually destroying the network
namespaces and restarting cleared the issue.

2014-10-10 10:18:29.452 13645 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for ff1c3f6e-21bd-4f47-988e-252c92a63aaa.
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp_agent.py", line 129, in call_driver
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     getattr(driver, action)(**action_kwargs)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 165, in enable
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     reuse_existing=True)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 826, in setup
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     namespace=network.namespace)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 178, in plug
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     namespace2=namespace)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 130, in add_veth
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     self._as_root('', 'link', tuple(args))
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 71, in _as_root
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     namespace)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 82, in _execute
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     root_helper=root_helper)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     raise RuntimeError(m)
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent RuntimeError:
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'add', 'tapefc7cb17-ce', 'type', 'veth', 'peer', 'name', 'ns-efc7cb17-ce', 'netns', 'qdhcp-ff1c3f6e-21bd-4f47-988e-252c92a63aaa']
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Exit code: 2
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Invalid argument\n'
2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent
2014-10-10 10:18:29.455 13645 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for ff9953b9-27b8-405b-bbd1-6be23f970f8c.
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp_agent.py", line 129, in call_driver
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     getattr(driver, action)(**action_kwargs)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 165, in enable
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     reuse_existing=True)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 826, in setup
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     namespace=network.namespace)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 178, in plug
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     namespace2=namespace)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 130, in add_veth
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     self._as_root('', 'link', tuple(args))
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 71, in _as_root
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     namespace)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 82, in _execute
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     root_helper=root_helper)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     raise RuntimeError(m)
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent RuntimeError:
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'add', 'tap46f808e2-04', 'type', 'veth', 'peer', 'name', 'ns-46f808e2-04', 'netns', 'qdhcp-ff9953b9-27b8-405b-bbd1-6be23f970f8c']
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Exit code: 2
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Invalid argument\n'
2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: neutron-common 1:2014.1.2-0ubuntu1.1
ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
Date: Fri Oct 10 10:22:21 2014
PackageArchitecture: all
SourcePackage: neutron
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.neutron.fwaas.driver.ini: [modified]
modified.conffile..etc.neutron.l3.agent.ini: [modified]
modified.conffile..etc.neutron.neutron.conf: [modified]
modified.conffile..etc.neutron.vpn.agent.ini: [modified]
mtime.conffile..etc.neutron.fwaas.driver.ini: 2014-10-10T10:00:04.304947
mtime.conffile..etc.neutron.l3.agent.ini: 2014-10-10T10:00:04.272947
mtime.conffile..etc.neutron.neutron.conf: 2014-10-10T10:00:04.076950
mtime.conffile..etc.neutron.vpn.agent.ini: 2014-10-10T10:00:04.224948

** Affects: neutron (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug trusty uec-images

** Description changed:

+ I saw this on upgrading to 2014.1.3; however downgrading was not
+ effective; stopping all daemons, manually destroying the network
+ namespaces and restarting cleared the issue.
+ 
  2014-10-10 10:18:29.452 13645 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for ff1c3f6e-21bd-4f47-988e-252c92a63aaa.
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp_agent.py", line 129, in call_driver
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     getattr(driver, action)(**action_kwargs)
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 165, in enable
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     reuse_existing=True)
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 826, in setup
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     namespace=network.namespace)
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 178, in plug
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     namespace2=namespace)
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 130, in add_veth
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     self._as_root('', 'link', tuple(args))
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 71, in _as_root
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     namespace)
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 82, in _execute
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     root_helper=root_helper)
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent     raise RuntimeError(m)
- 2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent RuntimeError: 
+ 2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent RuntimeError:
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'add', 'tapefc7cb17-ce', 'type', 'veth', 'peer', 'name', 'ns-efc7cb17-ce', 'netns', 'qdhcp-ff1c3f6e-21bd-4f47-988e-252c92a63aaa']
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Exit code: 2
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Stdout: ''
  2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Invalid argument\n'
- 2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent 
+ 2014-10-10 10:18:29.452 13645 TRACE neutron.agent.dhcp_agent
  2014-10-10 10:18:29.455 13645 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for ff9953b9-27b8-405b-bbd1-6be23f970f8c.
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp_agent.py", line 129, in call_driver
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     getattr(driver, action)(**action_kwargs)
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 165, in enable
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     reuse_existing=True)
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 826, in setup
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     namespace=network.namespace)
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/interface.py", line 178, in plug
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     namespace2=namespace)
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 130, in add_veth
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     self._as_root('', 'link', tuple(args))
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 71, in _as_root
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     namespace)
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 82, in _execute
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     root_helper=root_helper)
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 76, in execute
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent     raise RuntimeError(m)
- 2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent RuntimeError: 
+ 2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent RuntimeError:
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'add', 'tap46f808e2-04', 'type', 'veth', 'peer', 'name', 'ns-46f808e2-04', 'netns', 'qdhcp-ff9953b9-27b8-405b-bbd1-6be23f970f8c']
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Exit code: 2
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Stdout: ''
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Invalid argument\n'
  2014-10-10 10:18:29.455 13645 TRACE neutron.agent.dhcp_agent
  
  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: neutron-common 1:2014.1.2-0ubuntu1.1
  ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
  Uname: Linux 3.13.0-35-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.5
  Architecture: amd64
  Date: Fri Oct 10 10:22:21 2014
  PackageArchitecture: all
  SourcePackage: neutron
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.neutron.fwaas.driver.ini: [modified]
  modified.conffile..etc.neutron.l3.agent.ini: [modified]
  modified.conffile..etc.neutron.neutron.conf: [modified]
  modified.conffile..etc.neutron.vpn.agent.ini: [modified]
  mtime.conffile..etc.neutron.fwaas.driver.ini: 2014-10-10T10:00:04.304947
  mtime.conffile..etc.neutron.l3.agent.ini: 2014-10-10T10:00:04.272947
  mtime.conffile..etc.neutron.neutron.conf: 2014-10-10T10:00:04.076950
  mtime.conffile..etc.neutron.vpn.agent.ini: 2014-10-10T10:00:04.224948

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1379739

Title:
  Neutron daemons unable to manage network namespaces

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1379739/+subscriptions

More information about the Ubuntu-server-bugs mailing list