[Bug 1959098] Re: [ovn] metadata route missing on the guest

[Jose Guedez]

We hit this issue today as well. Same symptoms:

* Failed to get metadata during VM launch - consistently and only on the "affected" network. Other networks like "unaffected" are OK.
* Missing metadata route inside VM
* After adding the route manually to the .2 IP we can ping/curl the metadata endpoint with no issues, so it seems the route is the only thing missing.
* The workaround of adding the metadata route explicitly to the relevant router allows new VMs in the affected network to get metadata without problems.

These are the current packages:

ii neutron-common 2:16.4.2-0ubuntu1
ii neutron-ovn-metadata-agent 2:16.4.2-0ubuntu1
ii python3-neutron 2:16.4.2-0ubuntu1
ii python3-neutron-lib 2.3.0-0ubuntu1
ii python3-neutronclient 1:7.1.1-0ubuntu1

I am attaching the information requested above for an "affected" and
"unaffected" network. The main difference I see is that the "unaffected"
subnet has the following option in the ovn-nb that is missing from the
"affected" subnet:

classless_static_route="{169.254.169.254/32,10.131.83.2,
0.0.0.0/0,10.131.83.1}"

The two patches you mention are indeed included in python3-neutron
2:16.4.2-0ubuntu1. I additionally confirmed by checking
/usr/lib/python3/dist-
packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py

Regarding "ovn_metadata_enabled", I didn't find it set to "true" in any
config under /etc/neutron. I can only see the default commented out and
no mention in neutron_ovn_metadata_agent.ini, which has the ovs/ovn
config in it (but I am no expert)

/etc/neutron# grep -r ovn_metadata
ovn.ini:#ovn_metadata_enabled = false

The creation logs are no longer available. The ports for the .2 IPs are
created in the subnet, and they do have a device_id of
ovnmeta-<networkid>, but the device_owner is network:dhcp and not
network:distributed as you seem to be expecting. I added the output of
`port show` for them as well. Note that other networks on the same
compute nodes have no issues providing metadata, including the
"unaffected" network (data attached).

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1959098

Title:
  [ovn] metadata route missing on the guest

Status in neutron:
  New
Status in neutron package in Ubuntu:
  Confirmed

Bug description:
  * High level description

  Metadata server (169.254.169.254) is unreachable on VMs attached to only one affected network in the entire cluster. DHCP is enabled on that subnet and VMs get their IP addresses on boot, however the routing rule for metadata is missing:
  $ ip r
  default via 10.134.253.1 dev eth0 
  10.134.253.0/24 dev eth0 scope link  src 10.134.253.181 

  Because of that cloud-init metadata requests are being sent to the
  router rather than ovnmeta netns.

  On guests running in the unaffected network, routing table after booting or sending DHCP request looks like this and metadata endpoint is reachable:
  $ ip r
  default via 172.16.2.1 dev eth0 
  169.254.169.254 via 172.16.2.10 dev eth0 
  172.16.2.0/24 dev eth0 scope link  src 172.16.2.248 

  I managed to work this around by manually adding a route to the
  metadata IP via DHCP port on the router attached to that network,
  however I believe it should not be needed and such configuration is
  definitely not present on all the "good" networks on this cluster.

  Please let me know what logs and other information would be useful
  here.

  * Step-by-step reproduction steps

  1) Create a VM attached to the affected network.
  2) Metadata server is unreachable, cloud-init fails because of the missing route not being provided by DHCP server.

  * Expected output

  I'd expect metadata route to be present on the guest:

  $ ip r
  default via 10.134.253.1 dev eth0 
  169.254.169.254 via 10.134.253.2 dev eth0
  10.134.253.0/24 dev eth0 scope link  src 10.134.253.181 

  * Actual output:

  $ ip r
  default via 10.134.253.1 dev eth0 
  10.134.253.0/24 dev eth0 scope link  src 10.134.253.181 

  * Versions
  neutron-common             2:16.4.1-0ubuntu2
  neutron-ovn-metadata-agent 2:16.4.1-0ubuntu2
  python3-neutron            2:16.4.1-0ubuntu2
  python3-neutron-lib        2.3.0-0ubuntu1
  python3-neutronclient      1:7.1.1-0ubuntu1
  ovn-common                 20.03.2-0ubuntu0.20.04.1
  ovn-host                   20.03.2-0ubuntu0.20.04.1
  openvswitch-common         2.13.3-0ubuntu0.20.04.2
  openvswitch-switch         2.13.3-0ubuntu0.20.04.2
  python3-openvswitch        2.13.3-0ubuntu0.20.04.2
  python3-ovsdbapp           1.1.0-0ubuntu2

  Host OS: Ubuntu 20.04.3 LTS
  Kernel: 5.8.0-48-generic #54~20.04.1-Ubuntu
  Deployment: Juju charms

  Guest OS: cirros 0.5.2 and Ubuntu 20.04, so most likely all distros
  are affected

  * Environment

  42 compute nodes, nova-compute 21.2.2-0ubuntu1 + libvirt 6.0.0-0ubuntu8.14 + KVM.
  Deployed with Juju charms.

  * Perceived severity

  Not a blocker since there is a workaround.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1959098/+subscriptions