[Bug 1894843] Re: [dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host

[Dan Streetman]

** Description changed:

  [Impact]
  When neutron schedules snat namespaces it sometimes deletes the rfp interface from qrouter namespaces which breaks external network (fip) connectivity. The fix prevents this from happening.
  
  [Test Case]
   * deploy Openstack (Ussuri or above) with dvr_snat enabled in compute hosts.
   * ensure min. 2 compute hosts
   * create one ext network and one private network
   * add private subnet to router and ext as gateway
   * check which compute has the snat ns (ip netns| grep snat)
   * create a vm on each compute host
   * check that qrouter ns on both computes has rfp interface
   * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp
   * disable and re-enable router
   * openstack router set --disable <router>;  openstack router set --enable <router>
   * check again
   * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp
  
  [Where problems could occur]
- This patch is in fact restoring expected behaviour and is not expected to
- introduce any new regressions.
+ no regression is expected, but if one occurs it would likely result in breakage with external network connectivity
  
  -------------------------------------------------------------------------
  
  Hello,
  
  In the case of dvr_snat l3 agents are deployed on hypervisors there can
  be race condition. The agent creates snat namespaces on each scheduled
  host and removes them at second step. At this second step agent removes
  the rfp interface from qrouter even when there is VM with floating IP on
  the host.
  
  When VM is deployed at the time of second step we can lost external
  access to VMs floating IP. The issue can be reproduced by hand:
  
  1. Create tenant network and router with external gateway
  2. Create VM with floating ip
  3. Ensure that VM on the hypervisor without snat-* namespace
  4. Set the router to disabled state (openstack router set --disable <router>)
  5. Set the router to enabled state (openstack router set --enabled <router>)
  6. The external access to VMs FIP have lost because L3 agent creates the qrouter namespace without rfp interface.
  
  Environment:
  
  1. Neutron with ML2 OVS plugin.
  2. L3 agents in dvr_snat mode on each hypervisor
  3. openstack-neutron-common-15.1.1-0.20200611111910.7d97420.el8ost.noarch

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1894843

Title:
  [dvr_snat] Router update deletes rfp interface from qrouter even when
  VM port is present on this host

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Committed
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Triaged
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released

Bug description:
  [Impact]
  When neutron schedules snat namespaces it sometimes deletes the rfp interface from qrouter namespaces which breaks external network (fip) connectivity. The fix prevents this from happening.

  [Test Case]
   * deploy Openstack (Ussuri or above) with dvr_snat enabled in compute hosts.
   * ensure min. 2 compute hosts
   * create one ext network and one private network
   * add private subnet to router and ext as gateway
   * check which compute has the snat ns (ip netns| grep snat)
   * create a vm on each compute host
   * check that qrouter ns on both computes has rfp interface
   * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp
   * disable and re-enable router
   * openstack router set --disable <router>;  openstack router set --enable <router>
   * check again
   * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp

  [Where problems could occur]
  no regression is expected, but if one occurs it would likely result in breakage with external network connectivity

  -------------------------------------------------------------------------

  Hello,

  In the case of dvr_snat l3 agents are deployed on hypervisors there
  can be race condition. The agent creates snat namespaces on each
  scheduled host and removes them at second step. At this second step
  agent removes the rfp interface from qrouter even when there is VM
  with floating IP on the host.

  When VM is deployed at the time of second step we can lost external
  access to VMs floating IP. The issue can be reproduced by hand:

  1. Create tenant network and router with external gateway
  2. Create VM with floating ip
  3. Ensure that VM on the hypervisor without snat-* namespace
  4. Set the router to disabled state (openstack router set --disable <router>)
  5. Set the router to enabled state (openstack router set --enabled <router>)
  6. The external access to VMs FIP have lost because L3 agent creates the qrouter namespace without rfp interface.

  Environment:

  1. Neutron with ML2 OVS plugin.
  2. L3 agents in dvr_snat mode on each hypervisor
  3. openstack-neutron-common-15.1.1-0.20200611111910.7d97420.el8ost.noarch

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1894843/+subscriptions