[Bug 1891673] Fix merged to neutron (stable/stein)

Mon Jan 3 11:27:28 UTC 2022

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/810396
Committed: https://opendev.org/openstack/neutron/commit/d863a1cb7c88caf1eb63aa1c06c19e41aea7c581
Submitter: "Zuul (22348)"
Branch:    stable/stein

commit d863a1cb7c88caf1eb63aa1c06c19e41aea7c581
Author: Rodolfo Alonso Hernandez <ralonsoh at redhat.com>
Date:   Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Conflicts:
        neutron/tests/unit/agent/l3/test_dvr_local_router.py

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)
    (cherry picked from commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1891673

Title:
  qrouter ns ip rules not deleted when fip removed from vm

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Released
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released

Bug description:
  [Impact]

  neutron-l3-agent restart causes partial loss of fip information such
  that fip removal from vm results in ip rules left behind which breaks
  external network access for that vm.

  [Test Case]

  * deploy openstack with dvr enabled
  * create distributed router, network etc
  * create a vm and attach a floating ip
  * go to compute host on which vm is running and restart neutron-l3-agent
  * tail -f /var/log/neutron/neutron-l3-agent.log until it settles
  * remove fip from vm
  * run https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b on that compute node
  * should return with "nothing to do"

  [Regression Potential]

  the patch is reloading, on agent startup, information associated with
  floating ips, specifically the information needed to delete ip rules
  and rule priorities associated with a floating ip. Since that is
  essentially read-only I don't envisage a regression potential. When
  the l3-agent comes to use that information to delete the floating ip
  an error could occur if the information it is trying to delete no
  longer exists but that would not be a problem introduced by this patch
  so again, I don't envisage any potential for regressions from this
  patch since it doesn't change behavior in any way other than allowing
  the l3-agent to behave the same as if it hadn't been restarted.

  [Other Info]
  patched neutron l3 agent will reload info for *used* floating ips when restarted BUT if there are ip rules left behind from fips removed prior to using a pathed neutron then manual cleanup is still required and for that you can use https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b.

  --------------------------------------------------------------------------

  With Bionic Stein using dvr_snat if I add a floating ip to a vm then
  remove the floating ip, the corresponding ip rules in the associated
  qrouter ns local to the instance are not deleted which results in no
  longer being able to reach the external network because packets are
  still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute
  host running a vm whose address is 192.168.21.28 for which i have
  removed the fip I still see:

  # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list
  0:      from all lookup local
  32765:  from 192.168.21.28 lookup 16
  32766:  from all lookup main
  32767:  from all lookup default
  3232240897:     from 192.168.21.1/24 lookup 3232240897
  3232241231:     from 192.168.22.79/24 lookup 3232241231

  And table 16 leads to:

  # ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16
  default via 169.254.109.249 dev rfp-5e45608f-3

  Which results in the instance no longer being able to reach the
  external network (packets are never sent to the snat- ns in my case).

  The workaround is to delete that ip rule but neutron should be taking
  care of this. Looks like the culprit is in
  neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist

  Note that the NAT rules were successfully removed from iptables so
  looks like it is just this bit that is left behind.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1891673/+subscriptions