[Bug 1850779] Re: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart

[Corey Bryant]

A new version of neutron is now in the bionic unapproved queue awaiting
SRU team review:
https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=neutron

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1850779

Title:
  [L3] snat-ns will be initialized twice for DVR+HA routers during agent
  restart

Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Triaged
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released

Bug description:
  If the  DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart.
  And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent.

  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  SRU:

  [Impact]
  Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice.
  With this fix, the initialization phase is triggered only once.

  [Test Case]
  * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router
    (If stsstack-bundles are used, here are the commands
     ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run
     ./configure
     # Configure creates a router with external gateway attached
    )
  * Restart neutron-l3-agent on one of the node
    systemctl restart neutron-l3-agent.service

  * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps
    During initialization steps, following sysctl's are configured [1] [2]. 
    Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace.
    (If the fix is applied they should be displayed only once)
     
    grep -inr snat-<router-id> /var/log/neutron/neutron-l3-agent.log | grep sysctl

    Example log:
    2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87

  [Where problems could occur]
  This patch reduces the l3 agent initialization time during restart and is not expected to introduce new regressions.

  [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions