[Bug 1420572] [NEW] race between neutron-ovs-cleanup and nova-compute

[Launchpad Bug Tracker]

You have been subscribed to a public bug by Ubuntu Foundations Team Bug Bot (crichton):

[Impact]

 * We run neutron-ovs-cleanup in startup if neutron installed. If
   nova-compute does not wait for completion it will try to use
   veth/bridge devices that may be in the process of bring deleted.

[Test Case]

 * Create neutron (ovs) network and boot an instance with this network
   as --nic

 * Check that creation was successful and network is functional. Also make
   a note corresponding veth and bridge devices (ip a).

 * Reboot system, check that expected veth and bridge devices are still
   there and that nova-compute is happy e.g. try sshing to your instance.
   Also check /var/log/upstart/nova-compute.log to see if service waited
   for ovs-cleanup to finish.

[Regression Potential]

 * None

---- ---- ---- ----

There is a race when both neutron-ovs-cleanup and nova-compute trying to
do operations on the qvb*** and qvo*** devices. Below is a scenario I
recently met,

1. nova-compute was started and creating the veth_pair for VM instances
running on the host -
https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1298

2. neutron-ovs-cleanup was kicked off and deleted all the ports.

3. when nova-compute tried to set the MTU at
https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1280
, Stderr: u'Cannot find device "qvo***"\n' was reported. Because the
device that was just created was deleted again by neutron-ovs-cleanup.

As they both operate on the same resources, there needs a way to
synchronize the operations the two processes do on those resources.

** Affects: nova (Ubuntu)
     Importance: High
     Assignee: Edward Hope-Morley (hopem)
         Status: In Progress

** Affects: nova (Ubuntu Trusty)
     Importance: High
     Assignee: Edward Hope-Morley (hopem)
         Status: In Progress

** Affects: nova (Ubuntu Utopic)
     Importance: High
     Assignee: Edward Hope-Morley (hopem)
         Status: In Progress

** Affects: nova (Ubuntu Vivid)
     Importance: High
     Assignee: Edward Hope-Morley (hopem)
         Status: In Progress


** Tags: cts patch
-- 
race between neutron-ovs-cleanup and nova-compute
https://bugs.launchpad.net/bugs/1420572
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.