[Bug 1850779] Re: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart

[Hemanth Nakkina]

The test case is verified succesfully on bionic-proposed and sysctl
executions are done only once during neutron-l3-agent restart

** Tags removed: verification-needed verification-needed-bionic
** Tags added: verification-done verification-done-bionic

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1850779

Title:
  [L3] snat-ns will be initialized twice for DVR+HA routers during agent
  restart

Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Committed
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Groovy:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released

Bug description:
  If the  DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart.
  And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent.

  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  SRU:

  [Impact]
  Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice.
  With this fix, the initialization phase is triggered only once.

  [Test Case]
  * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router
    (If stsstack-bundles are used, here are the commands
     ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run
     ./configure
     # Configure creates a router with external gateway attached
    )
  * Restart neutron-l3-agent on one of the node
    systemctl restart neutron-l3-agent.service

  * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps
    During initialization steps, following sysctl's are configured [1] [2].
    Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace.
    (If the fix is applied they should be displayed only once)

    grep -inr snat-<router-id> /var/log/neutron/neutron-l3-agent.log |
  grep sysctl

    Example log:
    2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87

  [Where problems could occur]
  no regression is expected, but if one occurs it would likely result in longer init time and/or failure to correctly init the snat-namespace

  [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39
  [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions