[Bug 1630891] Re: unable to start lxd container instances after host reboot
James Page
james.page at ubuntu.com
Wed Nov 9 13:05:59 UTC 2016
Kevin - can you confirm versions please? This issue was quite specific
to OpenStack Newton - you might be seeing the same symptom but the
underlying cause may be different.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1630891
Title:
unable to start lxd container instances after host reboot
Status in Ubuntu Cloud Archive:
New
Status in nova-lxd:
Fix Released
Status in nova-lxd package in Ubuntu:
Confirmed
Bug description:
After a compute hypervisor is rebooted, instances revert to SHUTDOWN
state (which is the correct default); however, issuing a nova start
<uuid> does not start the instance up:
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 66, in wrapped
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 188, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 157, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 613, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2475, in start_instance
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self._power_on(context, instance)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2445, in _power_on
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server block_device_info)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 747, in power_on
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server container.start(wait=True)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 159, in start
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server wait=wait)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/container.py", line 144, in _set_state
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server self.client, response.json()['operation'])
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/operation.py", line 29, in wait_for_operation
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server operation.wait()
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/pylxd/models/operation.py", line 51, in wait
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server raise exceptions.LXDAPIException(response)
Oct 6 08:25:57 juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10 nova-compute[2958]: 2016-10-06 08:25:57.910 2958 ERROR oslo_messaging.rpc.server LXDAPIException: Missing parent 'qbrd853dc7e-2c' for nic 'qbrd853dc7e-2c'
It looks like the neutron network setup is not being re-created prior
to creating the container - this must be done by the compute driver
before attempting to start the instance:
root at juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether fa:16:3e:9d:8d:e1 brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether fa:8d:01:a5:24:0a brd ff:ff:ff:ff:ff:ff
4: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether 36:1b:99:1f:fd:4f brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether 26:e8:f0:7c:58:43 brd ff:ff:ff:ff:ff:ff
6: gre0 at NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/gre 0.0.0.0 brd 0.0.0.0
7: gretap0 at NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: gre_sys at NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 86:a0:50:4a:1b:b0 brd ff:ff:ff:ff:ff:ff
9: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether ae:20:44:00:f9:4b brd ff:ff:ff:ff:ff:ff
10: br-data: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
link/ether ee:3d:ef:52:b3:45 brd ff:ff:ff:ff:ff:ff
11: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 62:46:66:1a:af:04 brd ff:ff:ff:ff:ff:ff
root at juju-cf19f1e0-16f6-4889-8211-f9d4ec880eaf-machine-10:~# sudo ovs-vsctl show
86ce50a2-d54f-4d1a-92f2-655ff728d897
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "gre-0a051413"
Interface "gre-0a051413"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.5.20.25", out_key=flow, remote_ip="10.5.20.19"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-0a051412"
Interface "gre-0a051412"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.5.20.25", out_key=flow, remote_ip="10.5.20.18"}
Port br-tun
Interface br-tun
type: internal
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Bridge br-data
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-data
Interface br-data
type: internal
Port phy-br-data
Interface phy-br-data
type: patch
options: {peer=int-br-data}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port int-br-data
Interface int-br-data
type: patch
options: {peer=phy-br-data}
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
ovs_version: "2.6.0"
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nova-compute-lxd 14.0.0~rc1-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.4.0-38.57-generic 4.4.19
Uname: Linux 4.4.0-38-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Thu Oct 6 08:26:12 2016
Ec2AMI: ami-0000044a
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
TERM=screen-256color-bce
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: nova-lxd
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1630891/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list