[Bug 1420572] Re: race between neutron-ovs-cleanup and nova-compute

[Ubuntu Foundations Team Bug Bot]

The attachment "nova-compute-2014.1-lp1420572.patch" seems to be a
debdiff.  The ubuntu-sponsors team has been subscribed to the bug report
so that they can review and hopefully sponsor the debdiff.  If the
attachment isn't a patch, please remove the "patch" flag from the
attachment, remove the "patch" tag, and if you are member of the
~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by
~brian-murray, for any issue please contact him.]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1420572

Title:
  race between neutron-ovs-cleanup and nova-compute

Status in nova package in Ubuntu:
  In Progress
Status in nova source package in Trusty:
  In Progress
Status in nova source package in Utopic:
  In Progress
Status in nova source package in Vivid:
  In Progress

Bug description:
  [Impact]

   * We run neutron-ovs-cleanup in startup if neutron installed. If
     nova-compute does not wait for completion it will try to use
     veth/bridge devices that may be in the process of bring deleted.

  [Test Case]

   * Create neutron (ovs) network and boot an instance with this network
     as --nic

   * Check that creation was successful and network is functional. Also make
     a note corresponding veth and bridge devices (ip a).

   * Reboot system, check that expected veth and bridge devices are still
     there and that nova-compute is happy e.g. try sshing to your instance.
     Also check /var/log/upstart/nova-compute.log to see if service waited
     for ovs-cleanup to finish.

  [Regression Potential]

   * None

  ---- ---- ---- ----

  There is a race when both neutron-ovs-cleanup and nova-compute trying
  to do operations on the qvb*** and qvo*** devices. Below is a scenario
  I recently met,

  1. nova-compute was started and creating the veth_pair for VM
  instances running on the host -
  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1298

  2. neutron-ovs-cleanup was kicked off and deleted all the ports.

  3. when nova-compute tried to set the MTU at
  https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1280
  , Stderr: u'Cannot find device "qvo***"\n' was reported. Because the
  device that was just created was deleted again by neutron-ovs-cleanup.

  As they both operate on the same resources, there needs a way to
  synchronize the operations the two processes do on those resources.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572/+subscriptions