[Bug 1896938] Re: strange IPv6 NDP behaviour with OVN on Focal
Thomas Parrott
1896938 at bugs.launchpad.net
Thu Sep 24 12:50:00 UTC 2020
** Description changed:
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN
routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
- test
+ Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
+
+ # Install OVN and setup.
+ sudo apt install ovn-host ovn-central tcpdump -y
+
+ sudo ovs-vsctl set open_vswitch . \
+ external_ids:ovn-encap-type=geneve \
+ external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
+ external_ids:ovn-encap-ip=127.0.0.1
+
+ # Install LXD and get lxdbr0 address and subnet.
+ snap install lxd
+ lxd init --auto
+ lxc network show lxdbr0
+ config:
+ ipv4.address: 10.154.225.1/24
+ ipv4.nat: "true"
+ ipv6.address: fd42:37a6:2a0:c014::1/64
+ ipv6.nat: "true"
+
+ # Configure lxdbr0 as compatible OVN uplink network.
+ lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
+
+ # Create OVN network.
+ lxc network create ovn1 --type=ovn network=lxdbr0
+
+ # In separate window run tcpdump on lxsbr0.
+ sudo tcpdump -i lxdbr0 -n -e ip6
+
+ Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host,
+ e.g. fd42:37a6:2a0:c014::2
+
+ ping fd42:37a6:2a0:c014::2
+
+ # Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
+ 12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
+ 12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
+
+
+ # Create another OVN network.
+ lxc network create ovn2 --type=ovn network=lxdbr0
+
+
+ # Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
+ 12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
+ 12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
+ 12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
+
[Regression Potential]
[Other Info]
** Description changed:
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN
routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
- external_ids:ovn-encap-type=geneve \
- external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
- external_ids:ovn-encap-ip=127.0.0.1
+ external_ids:ovn-encap-type=geneve \
+ external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
+ external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
- config:
- ipv4.address: 10.154.225.1/24
- ipv4.nat: "true"
- ipv6.address: fd42:37a6:2a0:c014::1/64
- ipv6.nat: "true"
+ config:
+ ipv4.address: 10.154.225.1/24
+ ipv4.nat: "true"
+ ipv6.address: fd42:37a6:2a0:c014::1/64
+ ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
- # In separate window run tcpdump on lxsbr0.
+ # In separate window run tcpdump on lxdbr0 with -e flag to show MAC address.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host,
e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
-
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
-
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
-
[Regression Potential]
[Other Info]
** Description changed:
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN
routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxdbr0 with -e flag to show MAC address.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host,
e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
- # Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs
+ # Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs:
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
+ I would only expect to see the IPv6 NS packet from the LXD host's MAC
+ address, not the OVN gateways as well.
+
[Regression Potential]
[Other Info]
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1896938
Title:
strange IPv6 NDP behaviour with OVN on Focal
Status in ovn package in Ubuntu:
Fix Released
Status in ovn source package in Focal:
New
Status in ovn source package in Groovy:
Fix Released
Bug description:
[Impact]
The issue I'm seeing is that if I have multiple OVN routers connected to the same external OVS uplink switch (which in turn is connected via veth to a native linux bridge), when NDP solicit packets are sent from the native linux bridge into the logical OVN switch, for some reason every OVN router that is connected to the OVN switch retransmits the NDP packet from its own MAC address. Which in turn causes them all to receive another NDP packet, which are then all retransmitted (a flood then ensues until OVS rate limiting takes effect).
This occurs even for IPs that do not exist on the network that the OVN
routers dont know anything about.
It doesn't happen for ARP, nor does it occur in Groovy.
[Test Case]
Setup single node LXD with and OVN network and then ping a non-existant IP in the uplink network from the LXD host and watch for the OVN gateway to retransmit the IPv6 NS packet from lxdbr0 back into the uplink network.
# Install OVN and setup.
sudo apt install ovn-host ovn-central tcpdump -y
sudo ovs-vsctl set open_vswitch . \
external_ids:ovn-encap-type=geneve \
external_ids:ovn-remote="unix:/var/run/ovn/ovnsb_db.sock" \
external_ids:ovn-encap-ip=127.0.0.1
# Install LXD and get lxdbr0 address and subnet.
snap install lxd
lxd init --auto
lxc network show lxdbr0
config:
ipv4.address: 10.154.225.1/24
ipv4.nat: "true"
ipv6.address: fd42:37a6:2a0:c014::1/64
ipv6.nat: "true"
# Configure lxdbr0 as compatible OVN uplink network.
lxc network set lxdbr0 ipv4.dhcp.ranges=10.154.225.2-10.154.225.10 ipv4.ovn.ranges=10.154.225.11-10.154.225.20
# Create OVN network.
lxc network create ovn1 --type=ovn network=lxdbr0
# In separate window run tcpdump on lxdbr0 with -e flag to show MAC address.
sudo tcpdump -i lxdbr0 -n -e ip6
Now ping a non-existant IP in the subnet of lxdbr0 from the LXD host,
e.g. fd42:37a6:2a0:c014::2
ping fd42:37a6:2a0:c014::2
# Check tcpdump window showing duplicate IPv6 NS (from from lxdbr0 MAC address and one from the OVN network's external router port's MAC addres):
12:46:27.123496 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:46:27.124320 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
# Create another OVN network.
lxc network create ovn2 --type=ovn network=lxdbr0
# Check tcpdump window showing a duplicate IPv6 NS storm, now from 3 MACs:
12:48:30.025836 00:16:3e:b5:3d:5e > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028238 00:16:3e:ab:cc:5a > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
12:48:30.028239 00:16:3e:31:9a:ff > 33:33:ff:00:00:02, ethertype IPv6 (0x86dd), length 86: fd42:37a6:2a0:c014::1 > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has fd42:37a6:2a0:c014::2, length 32
I would only expect to see the IPv6 NS packet from the LXD host's MAC
address, not the OVN gateways as well.
[Regression Potential]
[Other Info]
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1896938/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list