[Bug 2169909] [NEW] [SRU] ovn 26.03.0 drops ARP requests from localnet to VMs on non-gateway chassis when the provider network has a router gateway (fixed in 26.03.2)
JayGue Lee
2169909 at bugs.launchpad.net
Wed Oct 7 23:26:01 UTC 2026
Public bug reported:
[ Impact ]
ovn 26.03.0 (Ubuntu Resolute 26.03.0-2, Ubuntu Cloud Archive gazpacho
26.03.0-2~cloud0) drops ARP requests that enter a logical switch through
its localnet port when the switch also has a distributed gateway router
port. In OpenStack terms this happens on any provider network that is
used as a Neutron router external gateway.
Upstream commit 1b4058b9 ("northd: Process external arps on ha chassis"),
which first shipped in v26.03.0 (it is not in v25.09.0), installs these
logical flows on such a switch:
ls_in_check_port_sec priority 75: arp.op == 1 && inport == "provnet-<id>" -> reg0[22] = 1
ls_in_apply_port_sec priority 85: reg0[22] == 1 && arp.tpa == <FIP> && is_chassis_resident("<fip-port>") -> next
ls_in_apply_port_sec priority 75: reg0[22] == 1 && is_chassis_resident("cr-lrp-<id>") -> next
ls_in_apply_port_sec priority 70: reg0[22] == 1 -> drop
On every chassis that is not the gateway chassis, an ARP request coming in
from the localnet port matches the priority 70 drop. As a result, a VM
plugged directly into the provider network on a non-gateway compute node
never receives ARP requests for its own IP from the physical network or
from VMs on other chassis.
The failure is limited to ARP requests (op=1) that arrive through
localnet. ARP requests sent by the VM, ARP replies (op=2) and normal IP
traffic are forwarded correctly. The VM port's port_security setting has
no effect: VMs with port security enabled are dropped as well.
User-visible effects:
* A VM on the provider network becomes unreachable from outside every
time the peer's ARP cache entry for it expires, until the VM itself
sends traffic that refreshes the entry.
* L2 VIPs that only answer ARP requests (for example Cilium L2
announcements or MetalLB in L2 mode) running inside such VMs are
unreachable permanently.
* VIP tools that send periodic gratuitous ARP (keepalived, kube-vip)
keep working.
Logical switches without a router port are not affected. In one
deployment with 5 provider logical switches, the 3 switches that have a
router port had the drop flow and the 2 without one did not.
Workaround: send a gratuitous ARP in reply form from the VM
("arping -U -P -i <iface> -S <ip> <ip>"). This restores reachability
until the peers' ARP cache entries expire again.
Fix: upstream reverted 1b4058b9 together with the 4 related commits that
came with it. The reverts are in v26.03.2 and v26.09.0 (v26.03.1 does not
include them). v26.03.2 also adds "northd: Enable ARP/ND responder for
localnet-sourced requests" and the test "ARP to VIF port on LS with
localnet port". This bug requests an update of ovn in Resolute and in the
gazpacho Ubuntu Cloud Archive to the 26.03.2 (or later 26.03.x) upstream
point release.
[ Test Plan ]
Minimal topology: one provider (localnet) network, one Neutron router
whose external gateway is on that network, and one VM on that network on
a compute node that is not the router's gateway chassis.
Example values used below: provider subnet 192.0.2.0/24, VM IP
192.0.2.10, external host 192.0.2.1 connected to the same physical
network, compute nodes compute-a (VM) and gateway node gw-a.
1. Create the topology:
openstack network create --external --provider-network-type flat \
--provider-physical-network physnet1 ext-net
openstack subnet create --network ext-net --subnet-range 192.0.2.0/24 \
--gateway 192.0.2.1 ext-subnet
openstack router create r1
openstack router set --external-gateway ext-net r1
openstack server create --network ext-net --flavor <flavor> \
--image <image> --availability-zone nova:compute-a vm1
Confirm that compute-a is not the gateway chassis of the router port:
ovn-nbctl lrp-get-gateway-chassis lrp-<router-gw-port-id>
2. Check (a), logical flows:
ovn-sbctl lflow-list neutron-<network-id> | grep 'reg0\[22\]'
Affected (26.03.0): the four flows listed in [ Impact ] are present,
including "priority=70, match=(reg0[22] == 1), action=(drop;)" in
ls_in_apply_port_sec.
3. Check (b), datapath behaviour:
From the external host:
arping -c 3 -I <iface> 192.0.2.10
Affected (26.03.0): no replies.
On compute-a, trace an ARP request entering from the localnet patch
port (for a VLAN provider network, add dl_vlan=<tag>):
ovs-vsctl get Interface patch-provnet-<id>-to-br-int ofport
ovs-appctl ofproto/trace br-int \
in_port=<ofport>,arp,arp_op=1,dl_src=<ext-mac>,dl_dst=ff:ff:ff:ff:ff:ff,arp_sha=<ext-mac>,arp_spa=192.0.2.1,arp_tha=00:00:00:00:00:00,arp_tpa=192.0.2.10
Affected (26.03.0) output, abbreviated:
8. arp,reg14=0x1,metadata=0x5,arp_op=1, priority 75 -> reg0[22]=1
9. reg0=0x400000/0x400000,metadata=0x5, priority 70 -> drop
The same trace with arp_op=2 reaches the VM's tap interface.
The drop flow counter on compute-a keeps increasing:
ovs-ofctl dump-flows br-int table=9 | grep 'reg0=0x400000/0x400000'
cookie=..., table=9, n_packets=57988, priority=70,reg0=0x400000/0x400000,metadata=0x5 actions=drop
The same happens between VMs: when a VM on another compute node sends
"who-has 192.0.2.10", the request reaches the uplink of compute-a but
is never delivered to vm1's tap interface.
4. Upgrade ovn on all nodes to the proposed package and repeat 2 and 3.
Expected (fixed): the "reg0[22] == 1 -> drop" flow is no longer
installed, the ofproto/trace for arp_op=1 reaches vm1's tap
interface, and arping from the external host gets replies.
5. Regression testing: run the usual OpenStack QA process for the UCA
(deployment plus Tempest smoke tests), including floating IP and
router gateway connectivity on a provider network.
[ Where problems could occur ]
* This is a minor upstream update from 26.03.0 to 26.03.2, so it brings
in all other fixes in 26.03.1 and 26.03.2, not only the reverts. Any
of these could change logical flow generation in northd or flow
installation in ovn-controller. A regression would show up as lost or
misrouted traffic on logical switches and routers after the update.
* The five reverted commits restore the 25.09 behaviour for ARP requests
that arrive from a localnet port on a switch with a distributed
gateway port. Deployments that came to depend on the 26.03.0 behaviour
for router gateway or dnat_and_snat (floating IP) addresses would see
ARP handling for those addresses change after the update.
* 26.03.2 adds a new ARP/ND responder for requests that come from a
localnet port. One possible side effect, which has NOT been verified,
is that the responder replies on behalf of a logical switch port that
is not bound to any chassis (status DOWN) but has the requested IP
reserved, using that port's MAC address. Deployments that reserve
IPs with unbound Neutron ports so that an L2 VIP inside a VM can
answer ARP for them would then see the VIP's traffic sent to the
reserved port's MAC instead. Testers should check this case
explicitly: create an unbound port with a fixed IP on the provider
network and confirm that ARP requests for that IP from the physical
network get no reply from OVN.
* During the upgrade, ovn-northd/NB/SB and ovn-controller run different
versions for a short time. Upgrade ordering should follow the usual
OVN guidance (ovn-controller first, then the central components).
[ Other Info ]
* Affected packages:
- Ubuntu Resolute (26.04): ovn 26.03.0-2
- Ubuntu Cloud Archive gazpacho (noble): ovn 26.03.0-2~cloud0
* Not affected: v25.09.0 (does not contain 1b4058b9).
* Cause:
https://github.com/ovn-org/ovn/commit/1b4058b9
Patch discussion:
http://www.mail-archive.com/ovs-dev@openvswitch.org/msg99363.html
* Upstream fix, in v26.03.2 and v26.09.0:
- 78659feb Revert "Process external arps on ha chassis"
- edf1f564 Revert "Fix receiving ARP reply on non-resident chassis"
- 759ec0da Revert "Fix for distributed dnat_and_snat ARP resolution"
- b218c635 Revert "Make sure we process external ARPs even on GW routers"
- d93cb42a Revert "Restrict external ARP request to logical_ip for dnat_and_snat"
Also in v26.03.2: "northd: Enable ARP/ND responder for localnet-sourced
requests" and the test "ARP to VIF port on LS with localnet port".
v26.03.1 does not contain the fix.
* Changelogs:
https://www.ovn.org/en/releases/changelog_v26.03.0/
https://www.ovn.org/en/releases/changelog_v26.03.2/
* OpenStack upgrades do not avoid the problem: kolla stable/2026.2
still uses OPENSTACK_RELEASE_CODENAME = 'Gazpacho' for its Ubuntu
images because no noble-updates/hibiscus pocket exists yet, so it
installs the same ovn 26.03.0 from the gazpacho UCA.
* Other distributions already ship the fix: the CentOS NFV SIG provides
ovn26.03-26.03.3 and ovn26.09-26.09.0.
* ovn is a supporting package under the OpenStack and Ubuntu Cloud
Archive SRU exception, so the update is requested as a new upstream
stable point release.
* Similar previous request: LP: #2168890.
* No existing Launchpad bug requests ovn 26.03.2 or later (checked
2026-10-07).
** Affects: cloud-archive
Importance: Undecided
Status: New
** Affects: ovn (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/2169909
Title:
[SRU] ovn 26.03.0 drops ARP requests from localnet to VMs on non-
gateway chassis when the provider network has a router gateway (fixed
in 26.03.2)
Status in Ubuntu Cloud Archive:
New
Status in ovn package in Ubuntu:
New
Bug description:
[ Impact ]
ovn 26.03.0 (Ubuntu Resolute 26.03.0-2, Ubuntu Cloud Archive gazpacho
26.03.0-2~cloud0) drops ARP requests that enter a logical switch through
its localnet port when the switch also has a distributed gateway router
port. In OpenStack terms this happens on any provider network that is
used as a Neutron router external gateway.
Upstream commit 1b4058b9 ("northd: Process external arps on ha chassis"),
which first shipped in v26.03.0 (it is not in v25.09.0), installs these
logical flows on such a switch:
ls_in_check_port_sec priority 75: arp.op == 1 && inport == "provnet-<id>" -> reg0[22] = 1
ls_in_apply_port_sec priority 85: reg0[22] == 1 && arp.tpa == <FIP> && is_chassis_resident("<fip-port>") -> next
ls_in_apply_port_sec priority 75: reg0[22] == 1 && is_chassis_resident("cr-lrp-<id>") -> next
ls_in_apply_port_sec priority 70: reg0[22] == 1 -> drop
On every chassis that is not the gateway chassis, an ARP request coming in
from the localnet port matches the priority 70 drop. As a result, a VM
plugged directly into the provider network on a non-gateway compute node
never receives ARP requests for its own IP from the physical network or
from VMs on other chassis.
The failure is limited to ARP requests (op=1) that arrive through
localnet. ARP requests sent by the VM, ARP replies (op=2) and normal IP
traffic are forwarded correctly. The VM port's port_security setting has
no effect: VMs with port security enabled are dropped as well.
User-visible effects:
* A VM on the provider network becomes unreachable from outside every
time the peer's ARP cache entry for it expires, until the VM itself
sends traffic that refreshes the entry.
* L2 VIPs that only answer ARP requests (for example Cilium L2
announcements or MetalLB in L2 mode) running inside such VMs are
unreachable permanently.
* VIP tools that send periodic gratuitous ARP (keepalived, kube-vip)
keep working.
Logical switches without a router port are not affected. In one
deployment with 5 provider logical switches, the 3 switches that have a
router port had the drop flow and the 2 without one did not.
Workaround: send a gratuitous ARP in reply form from the VM
("arping -U -P -i <iface> -S <ip> <ip>"). This restores reachability
until the peers' ARP cache entries expire again.
Fix: upstream reverted 1b4058b9 together with the 4 related commits that
came with it. The reverts are in v26.03.2 and v26.09.0 (v26.03.1 does not
include them). v26.03.2 also adds "northd: Enable ARP/ND responder for
localnet-sourced requests" and the test "ARP to VIF port on LS with
localnet port". This bug requests an update of ovn in Resolute and in the
gazpacho Ubuntu Cloud Archive to the 26.03.2 (or later 26.03.x) upstream
point release.
[ Test Plan ]
Minimal topology: one provider (localnet) network, one Neutron router
whose external gateway is on that network, and one VM on that network on
a compute node that is not the router's gateway chassis.
Example values used below: provider subnet 192.0.2.0/24, VM IP
192.0.2.10, external host 192.0.2.1 connected to the same physical
network, compute nodes compute-a (VM) and gateway node gw-a.
1. Create the topology:
openstack network create --external --provider-network-type flat \
--provider-physical-network physnet1 ext-net
openstack subnet create --network ext-net --subnet-range 192.0.2.0/24 \
--gateway 192.0.2.1 ext-subnet
openstack router create r1
openstack router set --external-gateway ext-net r1
openstack server create --network ext-net --flavor <flavor> \
--image <image> --availability-zone nova:compute-a vm1
Confirm that compute-a is not the gateway chassis of the router
port:
ovn-nbctl lrp-get-gateway-chassis lrp-<router-gw-port-id>
2. Check (a), logical flows:
ovn-sbctl lflow-list neutron-<network-id> | grep 'reg0\[22\]'
Affected (26.03.0): the four flows listed in [ Impact ] are present,
including "priority=70, match=(reg0[22] == 1), action=(drop;)" in
ls_in_apply_port_sec.
3. Check (b), datapath behaviour:
From the external host:
arping -c 3 -I <iface> 192.0.2.10
Affected (26.03.0): no replies.
On compute-a, trace an ARP request entering from the localnet patch
port (for a VLAN provider network, add dl_vlan=<tag>):
ovs-vsctl get Interface patch-provnet-<id>-to-br-int ofport
ovs-appctl ofproto/trace br-int \
in_port=<ofport>,arp,arp_op=1,dl_src=<ext-mac>,dl_dst=ff:ff:ff:ff:ff:ff,arp_sha=<ext-mac>,arp_spa=192.0.2.1,arp_tha=00:00:00:00:00:00,arp_tpa=192.0.2.10
Affected (26.03.0) output, abbreviated:
8. arp,reg14=0x1,metadata=0x5,arp_op=1, priority 75 -> reg0[22]=1
9. reg0=0x400000/0x400000,metadata=0x5, priority 70 -> drop
The same trace with arp_op=2 reaches the VM's tap interface.
The drop flow counter on compute-a keeps increasing:
ovs-ofctl dump-flows br-int table=9 | grep 'reg0=0x400000/0x400000'
cookie=..., table=9, n_packets=57988, priority=70,reg0=0x400000/0x400000,metadata=0x5 actions=drop
The same happens between VMs: when a VM on another compute node sends
"who-has 192.0.2.10", the request reaches the uplink of compute-a but
is never delivered to vm1's tap interface.
4. Upgrade ovn on all nodes to the proposed package and repeat 2 and
3.
Expected (fixed): the "reg0[22] == 1 -> drop" flow is no longer
installed, the ofproto/trace for arp_op=1 reaches vm1's tap
interface, and arping from the external host gets replies.
5. Regression testing: run the usual OpenStack QA process for the UCA
(deployment plus Tempest smoke tests), including floating IP and
router gateway connectivity on a provider network.
[ Where problems could occur ]
* This is a minor upstream update from 26.03.0 to 26.03.2, so it brings
in all other fixes in 26.03.1 and 26.03.2, not only the reverts. Any
of these could change logical flow generation in northd or flow
installation in ovn-controller. A regression would show up as lost or
misrouted traffic on logical switches and routers after the update.
* The five reverted commits restore the 25.09 behaviour for ARP requests
that arrive from a localnet port on a switch with a distributed
gateway port. Deployments that came to depend on the 26.03.0 behaviour
for router gateway or dnat_and_snat (floating IP) addresses would see
ARP handling for those addresses change after the update.
* 26.03.2 adds a new ARP/ND responder for requests that come from a
localnet port. One possible side effect, which has NOT been verified,
is that the responder replies on behalf of a logical switch port that
is not bound to any chassis (status DOWN) but has the requested IP
reserved, using that port's MAC address. Deployments that reserve
IPs with unbound Neutron ports so that an L2 VIP inside a VM can
answer ARP for them would then see the VIP's traffic sent to the
reserved port's MAC instead. Testers should check this case
explicitly: create an unbound port with a fixed IP on the provider
network and confirm that ARP requests for that IP from the physical
network get no reply from OVN.
* During the upgrade, ovn-northd/NB/SB and ovn-controller run different
versions for a short time. Upgrade ordering should follow the usual
OVN guidance (ovn-controller first, then the central components).
[ Other Info ]
* Affected packages:
- Ubuntu Resolute (26.04): ovn 26.03.0-2
- Ubuntu Cloud Archive gazpacho (noble): ovn 26.03.0-2~cloud0
* Not affected: v25.09.0 (does not contain 1b4058b9).
* Cause:
https://github.com/ovn-org/ovn/commit/1b4058b9
Patch discussion:
http://www.mail-archive.com/ovs-dev@openvswitch.org/msg99363.html
* Upstream fix, in v26.03.2 and v26.09.0:
- 78659feb Revert "Process external arps on ha chassis"
- edf1f564 Revert "Fix receiving ARP reply on non-resident chassis"
- 759ec0da Revert "Fix for distributed dnat_and_snat ARP resolution"
- b218c635 Revert "Make sure we process external ARPs even on GW routers"
- d93cb42a Revert "Restrict external ARP request to logical_ip for dnat_and_snat"
Also in v26.03.2: "northd: Enable ARP/ND responder for localnet-sourced
requests" and the test "ARP to VIF port on LS with localnet port".
v26.03.1 does not contain the fix.
* Changelogs:
https://www.ovn.org/en/releases/changelog_v26.03.0/
https://www.ovn.org/en/releases/changelog_v26.03.2/
* OpenStack upgrades do not avoid the problem: kolla stable/2026.2
still uses OPENSTACK_RELEASE_CODENAME = 'Gazpacho' for its Ubuntu
images because no noble-updates/hibiscus pocket exists yet, so it
installs the same ovn 26.03.0 from the gazpacho UCA.
* Other distributions already ship the fix: the CentOS NFV SIG provides
ovn26.03-26.03.3 and ovn26.09-26.09.0.
* ovn is a supporting package under the OpenStack and Ubuntu Cloud
Archive SRU exception, so the update is requested as a new upstream
stable point release.
* Similar previous request: LP: #2168890.
* No existing Launchpad bug requests ovn 26.03.2 or later (checked
2026-10-07).
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2169909/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list