[Bug 1927868] Re: vRouter not working after update to 16.3.1

[LIU Yulong]

Seems to be a terrible bug... I have a question about juju managed
Openstack, it will directly use the tag from upstream neutron without
any private pathes or backports? If so, the following is the patches
from 13.6.0 ahead to 13.6.1, maybe we can found the problem from it.

7771f16 [L3] Delete DvrFipGatewayPortAgentBindings after no gw ports
7f3aadd Lock sg updates while in _apply_port_filter()
9736efd [OVN] Set mcast_flood_reports on LSPs
ebc9921 Revert "DVR: Remove control plane arp updates for DVR"
9539db1 Add minimum bw qos rule validation for network
f8f1eaf Add some wait time between stopping and starting again ovsdb monitor
b16c29f Ignore python warnings in the fullstack job
590551d [L3][HA] Retry when setting HA router GW status.
664ee1d Fix wrong packet_type set for IPv6 GRE tunnels in OVS
630fc3b Stop metadata proxy gracefully
28cf678 Delete HA metadata proxy PID and config with elevated privileges
2f4ef31 Improve "get_devices_with_ip" performance
580e57b [OVS FW] Allow egress ICMPv6 only for know addresses
0c75fd0 Remove update_initial_state() method from the HA router
87fce78 Migrate "netstat" to oslo.privsep
817c5f2 [OVS FW] Clean conntrack entries with mark == CT_MARK_INVALID
73e1672 Fix deletion of rfp interfaces when router is re-enabled
7ce3c8e Avoid race condition when processing RowEvents
b2dc70e [OVN] ovn-metadata-agent: Retry registering Chassis at startup
f39230d Fix update of trunk subports during live migration
75b8fa7 Add extension unit tests for conntrack_helper plugin
9139f40 Fix incorrect exception catch when update floating ip port forwarding
f849a4c Don't try to create default SG when security groups are disabled
dc01f5b Process DHCP events in order if related
55a82da [OVN] Update metadata port ony for requested subnet
b84dbd6 Fix losses of ovs flows when ovs is restarted
5f78ff5 Make test_agent_show only look for its own agents
c825921 Do not update agents "alive" state in TestAgentApi
76a57e9 Optimize get_ports with QoS extension
a76a3f2 [OVN] Ensure metadata checksum
a2f312d Auto-remove floating agent gw ports on net/subnet delete
1f71ae2 [QoS] Get only min bw rules when extending port dict
e24a66d Optimize get_ports with trunk extension
8cb6130 Improve DHCP agent's debug messages
f7e028b ovn: Support live migration to DPDK nodes
daba68d Add WaitForPortCreateEvent in BaseOVSTestCase

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1927868

Title:
  vRouter not working after update to 16.3.1

Status in neutron:
  New
Status in neutron package in Ubuntu:
  Confirmed

Bug description:
  We run a juju managed Openstack Ussuri on Bionic. After updating
  neutron packages from 16.3.0 to 16.3.1 all virtual routers stopped
  working. It seems that most (not all) namespaces are created but have
  only the lo interface and sometime the ha-XYZ interface in DOWN state.
  The underlying tap interfaces are also in down.

  neutron-l3-agent has many logs similar to the following:
  2021-05-08 15:01:45.286 39411 ERROR neutron.agent.l3.ha_router [-] Gateway interface for router 02945b59-639b-41be-8237-3b7933b4e32d was not set up; router will not work properly

  and journal logs report at around the same time
  May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.765 18596 INFO neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 62.62.62.62 on qg-5a6efe8c-6b in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d: Exit code: 2; Stdin: ; Stdout: Interface "qg-5a6efe8c-6b" is down
  May 08 15:01:40 lar1615.srv-louros.grnet.gr neutron-keepalived-state-change[18596]: 2021-05-08 15:01:40.767 18596 INFO neutron.agent.linux.ip_lib [-] Interface qg-5a6efe8c-6b or address 62.62.62.62 in namespace qrouter-02945b59-639b-41be-8237-3b7933b4e32d was deleted concurrently

  
  The neutron packages installed are:

  ii  neutron-common                         2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - common
  ii  neutron-dhcp-agent                     2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - DHCP agent
  ii  neutron-l3-agent                       2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - l3 agent
  ii  neutron-metadata-agent                 2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - metadata agent
  ii  neutron-metering-agent                 2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - metering agent
  ii  neutron-openvswitch-agent              2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
  ii  python3-neutron                        2:16.3.1-0ubuntu1~cloud0                                    all          Neutron is a virtual network service for Openstack - Python library
  ii  python3-neutron-lib                    2.3.0-0ubuntu1~cloud0                                       all          Neutron shared routines and utilities - Python 3.x
  ii  python3-neutronclient                  1:7.1.1-0ubuntu1~cloud0                                     all          client API library for Neutron - Python 3.x


  Downgrading to 16.3.0 resolves the issues.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1927868/+subscriptions