[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
Frode Nordahl
1967856 at bugs.launchpad.net
Mon Jul 18 20:18:05 UTC 2022
** Description changed:
- If you have two hvs where hv1 is the gateway chassis and you have an
- instance running on hv2.
+ [Impact]
+ Users of Open vSwitch on Focal will not be able to upgrade to v2.16.0 or newer until this long standing kernel bug has been fixed.
+
+ Users of Open vSwitch on Jammy will be affected by this bug and
+ have no user space fix available. This bug currently blocks the
+ OpenStack Engineering team's charm product gate.
+
+ [Test Plan]
+ Execute the OVN system testsuite utilizing the kernel data path with the test synthesis patch in comment #7 applied.
+
+ In addition to that validating that the OpenStack charm test gate is
+ unblocked would be valuable.
+
+ [Regression Potential]
+ The regression potential can be considered as low because:
+ - The calls added in the openvswitch kernel datapath code would
+ prior to Open vSwitch 2.16.0 have been initiated from the
+ userspace code and by chance concealed this bug.
+ - After an optimization done in 2.16.0 the kernel bug was
+ revealed and these calls now must be made from the kernel
+ datapath to retain functionality in use in the wild.
+
+ [Original Bug Description]
+ If you have two hvs where hv1 is the gateway chassis and you have an instance running on hv2.
On instance on hv2 hairpin traffic works for the first session, but not
for the next:
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.07 ms
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.078/1.078/1.078/0.000 ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7
-
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7335,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7335,type=0,code=0),zone=7
We made an attempt at using OVN built with [0], but that did
unfortunately not help.
If we however revert [1] it works again:
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.31 ms
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.318/1.318/1.318/0.000 ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=0.307 ms
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.307/0.307/0.307/0.000 ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=1
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1
-
0: https://patchwork.ozlabs.org/project/ovn/patch/20220401175516.2139179-1-mmichels@redhat.com/
1: https://github.com/ovn-org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
- ---
+ ---
ProblemType: Bug
AlsaDevices:
- total 0
- crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq
- crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer
+ total 0
+ crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq
+ crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
- Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
- Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+ Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
+ Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
- /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
- /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
+ /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
+ /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
Package: linux (not installed)
PciMultimedia:
-
+
ProcEnviron:
- TERM=xterm-256color
- PATH=(custom, no user)
- LANG=C.UTF-8
- SHELL=/bin/bash
+ TERM=xterm-256color
+ PATH=(custom, no user)
+ LANG=C.UTF-8
+ SHELL=/bin/bash
ProcFB: 0 virtio_gpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-37-generic root=UUID=63713e6b-8e8d-4f97-ac5a-883317b24711 ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35
RelatedPackageVersions:
- linux-restricted-modules-5.15.0-37-generic N/A
- linux-backports-modules-5.15.0-37-generic N/A
- linux-firmware 20220329.git681281e4-0ubuntu1
+ linux-restricted-modules-5.15.0-37-generic N/A
+ linux-backports-modules-5.15.0-37-generic N/A
+ linux-firmware 20220329.git681281e4-0ubuntu1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/06/2015
dmi.bios.release: 0.0
dmi.bios.vendor: EFI Development Kit II / OVMF
dmi.bios.version: 0.0.0
dmi.board.name: LXD
dmi.board.vendor: Canonical Ltd.
dmi.board.version: pc-q35-7.0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-7.0
dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-7.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-7.0:cvnQEMU:ct1:cvrpc-q35-7.0:sku:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-7.0
dmi.sys.vendor: QEMU
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1967856
Title:
Hairpin traffic does not work with centralized NAT gw
Status in linux package in Ubuntu:
Invalid
Status in openvswitch package in Ubuntu:
Invalid
Status in ovn package in Ubuntu:
Invalid
Status in linux source package in Focal:
Fix Committed
Status in openvswitch source package in Focal:
Invalid
Status in ovn source package in Focal:
Invalid
Status in linux source package in Impish:
Won't Fix
Status in openvswitch source package in Impish:
New
Status in ovn source package in Impish:
New
Status in linux source package in Jammy:
Fix Committed
Status in openvswitch source package in Jammy:
Invalid
Status in ovn source package in Jammy:
Invalid
Status in linux source package in Kinetic:
Invalid
Status in openvswitch source package in Kinetic:
Invalid
Status in ovn source package in Kinetic:
Invalid
Bug description:
[Impact]
Users of Open vSwitch on Focal will not be able to upgrade to v2.16.0 or newer until this long standing kernel bug has been fixed.
Users of Open vSwitch on Jammy will be affected by this bug and
have no user space fix available. This bug currently blocks the
OpenStack Engineering team's charm product gate.
[Test Plan]
Execute the OVN system testsuite utilizing the kernel data path with the test synthesis patch in comment #7 applied.
In addition to that validating that the OpenStack charm test gate is
unblocked would be valuable.
[Regression Potential]
The regression potential can be considered as low because:
- The calls added in the openvswitch kernel datapath code would
prior to Open vSwitch 2.16.0 have been initiated from the
userspace code and by chance concealed this bug.
- After an optimization done in 2.16.0 the kernel bug was
revealed and these calls now must be made from the kernel
datapath to retain functionality in use in the wild.
[Original Bug Description]
If you have two hvs where hv1 is the gateway chassis and you have an instance running on hv2.
On instance on hv2 hairpin traffic works for the first session, but
not for the next:
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.07 ms
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.078/1.078/1.078/0.000 ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7334,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7334,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7335,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7335,type=0,code=0),zone=7
We made an attempt at using OVN built with [0], but that did
unfortunately not help.
If we however revert [1] it works again:
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=1.31 ms
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.318/1.318/1.318/0.000 ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1
$ ping -c1 10.78.95.89
PING 10.78.95.89 (10.78.95.89) 56(84) bytes of data.
64 bytes from 10.78.95.89: icmp_seq=1 ttl=62 time=0.307 ms
--- 10.78.95.89 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.307/0.307/0.307/0.000 ms
$ sudo ovs-appctl -t ovs-vswitchd dpctl/dump-conntrack
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=0,code=0),zone=1
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7337,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7337,type=0,code=0),zone=7
icmp,orig=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=8,code=0),reply=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=7
icmp,orig=(src=10.78.95.89,dst=192.168.0.211,id=7336,type=8,code=0),reply=(src=192.168.0.211,dst=10.78.95.89,id=7336,type=0,code=0),zone=1
0: https://patchwork.ozlabs.org/project/ovn/patch/20220401175516.2139179-1-mmichels@redhat.com/
1: https://github.com/ovn-org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jun 9 11:35 seq
crw-rw---- 1 root audio 116, 33 Jun 9 11:35 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
MachineType: QEMU Standard PC (Q35 + ICH9, 2009)
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB: 0 virtio_gpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-37-generic root=UUID=63713e6b-8e8d-4f97-ac5a-883317b24711 ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35
RelatedPackageVersions:
linux-restricted-modules-5.15.0-37-generic N/A
linux-backports-modules-5.15.0-37-generic N/A
linux-firmware 20220329.git681281e4-0ubuntu1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy uec-images
Uname: Linux 5.15.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/06/2015
dmi.bios.release: 0.0
dmi.bios.vendor: EFI Development Kit II / OVMF
dmi.bios.version: 0.0.0
dmi.board.name: LXD
dmi.board.vendor: Canonical Ltd.
dmi.board.version: pc-q35-7.0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-q35-7.0
dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(Q35+ICH9,2009):pvrpc-q35-7.0:rvnCanonicalLtd.:rnLXD:rvrpc-q35-7.0:cvnQEMU:ct1:cvrpc-q35-7.0:sku:
dmi.product.name: Standard PC (Q35 + ICH9, 2009)
dmi.product.version: pc-q35-7.0
dmi.sys.vendor: QEMU
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1967856/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list