[Bug 1927868] Re: vRouter not working after update to 16.3.1

[Edward Hope-Morley]

I've had a go at deploying Train and upgrading Neutron to latest Ussuri
and I see the same issue. Looking closer what I see is that post-upgrade
Neutron l3-agent has not spawned any keepalived processes hence why no
router goes active. When the agent is restarted it would normally
receive two router updates; first one to spawn_state_change_monitor and
a second to spawn keepalived. In my non-working nodes the second router
update is never received by the l3-agent. Here is an example of a
working agent https://pastebin.ubuntu.com/p/PFb594wkhB vs. a not working
https://pastebin.ubuntu.com/p/MtDNrXmvZB/.

I tested restarted all agents and this did not fix things. I then
rebooted one of my upgraded nodes and it resolved the issue for that
node i.e. two updates received and both spawned then router goes active.
I also noticed that on a non-rebooted node, following ovs agent restart
I see https://pastebin.ubuntu.com/p/2n4KxBv8S2/ which again is not
resolved by an agent restart and is fixed by the node reboot. This
latter issue is described on old bugs e.g.
https://bugs.launchpad.net/neutron/+bug/1625305

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1927868

Title:
  vRouter not working after update to 16.3.1

Status in neutron:
  New
Status in neutron package in Ubuntu:
  New

Bug description:
  We run a juju managed Openstack Ussuri on Bionic. After updating
  neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped
  working. It seems that most (not all) namespaces are created but have
  only the lo interface and sometime the ha-XYZ interface in DOWN state.
  The underlying tap interfaces are also in down.

  neutron-l3-agent has many logs similar to the following:
  2021-05-08 15:01:45.286 39411 ERROR neutron.agent.l3.ha_router [-] Gateway interface for router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will not work properly

  and journal logs report at around the same time
  May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.765 18596 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down
  May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.767 18596 INFO neutron.agent.linux.ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d was deleted concurrently

  
  The neutron packages installed are:

  ii  neutron-common                         2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - common
  ii  neutron-dhcp-agent                     2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - DHCP agent
  ii  neutron-l3-agent                       2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - l3 agent
  ii  neutron-metadata-agent                 2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - metadata agent
  ii  neutron-metering-agent                 2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - metering agent
  ii  neutron-openvswitch-agent              2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
  ii  python3-neutron                        2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - Python library
  ii  python3-neutron-lib                    2.3.0-0ubuntu1~cloud0                                       all          Neutron shared routines and utilities - Python 3.x
  ii  python3-neutronclient                  1:7.1.1-0ubuntu1~cloud0                                     all          client API library for Neutron - Python 3.x


  Downgrading to 16.3.0 resolves the issues.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1927868/+subscriptions