[Bug 1927868] Re: vRouter not working after update to 16.3.1

[OpenStack Infra]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/809219
Committed: https://opendev.org/openstack/neutron/commit/b3a70fe75315fba061ecb7d6ac1d50a04768ec13
Submitter: "Zuul (22348)"
Branch:    stable/wallaby

commit b3a70fe75315fba061ecb7d6ac1d50a04768ec13
Author: Edward Hope-Morley <edward.hope-morley at canonical.com>
Date:   Fri Aug 20 12:25:04 2021 +0100

    Revert "[L3][HA] Retry when setting HA router GW status."
    
    In short this patch can cause the privsep reader thread to
    die resulting in the l3 agent getting stuck and e.g. not
    processing any router updates. See related LP bug for full
    explanation.
    
    Closes-Bug: #1927868
    
    This reverts commit 662f483120972a373e19bde52f16392e2ccb9c82.
    
    Change-Id: Ide7e9771d08eb623dd75941e425813d9b857b4c6
    (cherry picked from commit 344fc0c8d2ce7d942606c834a54cb81f0b47aa37)


** Tags added: in-stable-wallaby

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1927868

Title:
  vRouter not working after update to 16.3.1

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in Ubuntu Cloud Archive victoria series:
  Fix Released
Status in Ubuntu Cloud Archive wallaby series:
  Fix Released
Status in Ubuntu Cloud Archive xena series:
  Fix Released
Status in neutron:
  Fix Released
Status in oslo.privsep:
  New
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Focal:
  Fix Released
Status in neutron source package in Hirsute:
  Fix Released
Status in neutron source package in Impish:
  Fix Released

Bug description:
  We run a juju managed Openstack Ussuri on Bionic. After updating
  neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped
  working. It seems that most (not all) namespaces are created but have
  only the lo interface and sometime the ha-XYZ interface in DOWN state.
  The underlying tap interfaces are also in down.

  neutron-l3-agent has many logs similar to the following:
  2021-05-08 15:01:45.286 39411 ERROR neutron.agent.l3.ha_router [-] Gateway interface for router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will not work properly

  and journal logs report at around the same time
  May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.765 18596 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down
  May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.767 18596 INFO neutron.agent.linux.ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d was deleted concurrently

  The neutron packages installed are:

  ii  neutron-common                         2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - common
  ii  neutron-dhcp-agent                     2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - DHCP agent
  ii  neutron-l3-agent                       2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - l3 agent
  ii  neutron-metadata-agent                 2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - metadata agent
  ii  neutron-metering-agent                 2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - metering agent
  ii  neutron-openvswitch-agent              2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
  ii  python3-neutron                        2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - Python library
  ii  python3-neutron-lib                    2.3.0-0ubuntu1~cloud0                                       all          Neutron shared routines and utilities - Python 3.x
  ii  python3-neutronclient                  1:7.1.1-0ubuntu1~cloud0                                     all          client API library for Neutron - Python 3.x

  Downgrading to 16.3.0 resolves the issues.

  =================================

  Ubuntu SRU details:

  [Impact]
  See above.

  [Test Case]
  Deploy openstack with l3ha and create several HA routers, the number required varies per environment. It is probably best to deploy a known bad version of the package, ensure it is failing, upgrade to the version in proposed, and re-test several times to confirm it is fixed.

  Restarting neutron-l3-agent should expect all HA Routers are restored.

  [Regression Potential]
  This change is fixing a regression by reverting a patch that was introduced in a stable point release of neutron.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1927868/+subscriptions