[Bug 1959098] Re: [ovn] metadata route missing on the guest

[yatin]

*** This bug is a duplicate of bug 2015377 ***
    https://bugs.launchpad.net/bugs/2015377

So i crossed this again, and looking to the shared output bits i see:-

affected subnet created at 2022-04-11T05:57:38Z

metadata port created at 2022-04-26T18:02:38Z (15 days later)

Couldn't know what created that port without logs, it's likely created
manually or with ovn-db sync utility. In Ussuri 16.4.2 network create
should create that port and subnet create should update that port along
with assigning ip and updating metadata routes in dhcp_options. But if
metadata port is missing during subnet create then metadata port do not
get recreated and thus missing routes in dhcp_options.

With https://review.opendev.org/c/openstack/neutron/+/883508 which is
fixed after neutron-16.4.2 handles missing metadata port during subnet
create or update. Based on these symptoms will mark this as duplicate of
https://bugs.launchpad.net/neutron/+bug/2015377. Feel free to reopen if
it happens even with the linked fixes on other bug.

** This bug has been marked a duplicate of bug 2015377
   [ovn] If dhcp port is deleted from neutron, it is never recreated

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1959098

Title:
  [ovn] metadata route missing on the guest

Status in neutron:
  New
Status in neutron package in Ubuntu:
  Confirmed

Bug description:
  * High level description

  Metadata server (169.254.169.254) is unreachable on VMs attached to only one affected network in the entire cluster. DHCP is enabled on that subnet and VMs get their IP addresses on boot, however the routing rule for metadata is missing:
  $ ip r
  default via 10.134.253.1 dev eth0 
  10.134.253.0/24 dev eth0 scope link  src 10.134.253.181 

  Because of that cloud-init metadata requests are being sent to the
  router rather than ovnmeta netns.

  On guests running in the unaffected network, routing table after booting or sending DHCP request looks like this and metadata endpoint is reachable:
  $ ip r
  default via 172.16.2.1 dev eth0 
  169.254.169.254 via 172.16.2.10 dev eth0 
  172.16.2.0/24 dev eth0 scope link  src 172.16.2.248 

  I managed to work this around by manually adding a route to the
  metadata IP via DHCP port on the router attached to that network,
  however I believe it should not be needed and such configuration is
  definitely not present on all the "good" networks on this cluster.

  Please let me know what logs and other information would be useful
  here.

  * Step-by-step reproduction steps

  1) Create a VM attached to the affected network.
  2) Metadata server is unreachable, cloud-init fails because of the missing route not being provided by DHCP server.

  * Expected output

  I'd expect metadata route to be present on the guest:

  $ ip r
  default via 10.134.253.1 dev eth0 
  169.254.169.254 via 10.134.253.2 dev eth0
  10.134.253.0/24 dev eth0 scope link  src 10.134.253.181 

  * Actual output:

  $ ip r
  default via 10.134.253.1 dev eth0 
  10.134.253.0/24 dev eth0 scope link  src 10.134.253.181 

  * Versions
  neutron-common             2:16.4.1-0ubuntu2
  neutron-ovn-metadata-agent 2:16.4.1-0ubuntu2
  python3-neutron            2:16.4.1-0ubuntu2
  python3-neutron-lib        2.3.0-0ubuntu1
  python3-neutronclient      1:7.1.1-0ubuntu1
  ovn-common                 20.03.2-0ubuntu0.20.04.1
  ovn-host                   20.03.2-0ubuntu0.20.04.1
  openvswitch-common         2.13.3-0ubuntu0.20.04.2
  openvswitch-switch         2.13.3-0ubuntu0.20.04.2
  python3-openvswitch        2.13.3-0ubuntu0.20.04.2
  python3-ovsdbapp           1.1.0-0ubuntu2

  Host OS: Ubuntu 20.04.3 LTS
  Kernel: 5.8.0-48-generic #54~20.04.1-Ubuntu
  Deployment: Juju charms

  Guest OS: cirros 0.5.2 and Ubuntu 20.04, so most likely all distros
  are affected

  * Environment

  42 compute nodes, nova-compute 21.2.2-0ubuntu1 + libvirt 6.0.0-0ubuntu8.14 + KVM.
  Deployed with Juju charms.

  * Perceived severity

  Not a blocker since there is a workaround.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1959098/+subscriptions