[Bug 1966981] Re: [hwol] centralized gateway NAT mangles source address
Frode Nordahl
1966981 at bugs.launchpad.net
Thu Mar 31 06:05:26 UTC 2022
** Description changed:
When hardware offload is enabled, CMS configures gateway routers and
centralized NAT, the source address of packets flowing through the
gateway are mangled after the connection preamble, not allowing the
connection to form. Return traffic on connections established from the
instance are also affected, even for UDP (DNS).
Example outside-in TCP connection attempt:
tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet), capture size 262144 bytes
05:52:04.286472 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 8865, offset 0, flags [DF], proto TCP (6), length 60)
10.11.2.11.57704 > 10.42.3.34.22: Flags [S], cksum 0x1e2a (correct), seq 3991821787, win 64240, options [mss 1460,sackOK,TS val 943370240 ecr 0,nop,wscale 7], length 0
05:52:04.286661 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.42.3.34.22 > 10.11.2.11.57704: Flags [S.], cksum 0x1990 (incorrect -> 0xdac4), seq 2903091535, ack 3991821788, win 62230, options [mss 8902,sackOK,TS val 2422032229 ecr 943370240,nop,wscale 7], length 0
05:52:04.287314 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 62, id 8866, offset 0, flags [DF], proto TCP (6), length 52)
10.11.2.11.57704 > 10.42.3.34.22: Flags [.], cksum 0x17c3 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 943370241 ecr 2422032229], length 0
05:52:04.287589 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 107: (tos 0x0, ttl 62, id 8867, offset 0, flags [DF], proto TCP (6), length 93)
0.0.0.0.57704 > 10.42.3.34.22: Flags [P.], cksum 0x017f (correct), seq 3991821788:3991821829, ack 2903091536, win 502, options [nop,nop,TS val 943370242 ecr 2422032229], length 41
Return traffic on connection-less UDP streams are also affected:
05:51:56.441418 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 56915, offset 0, flags [DF], proto UDP (17), length 90)
10.42.3.34.57502 > 194.169.254.1.53: [bad udp cksum 0xce4e -> 0x88d1!] 38125+ AAAA? fnordahl-bastion.openstack.partnercloud1.lan. (62)
05:51:56.443387 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 179: (tos 0x0, ttl 63, id 64535, offset 0, flags [DF], proto UDP (17), length 165)
0.0.0.0.53 > 10.42.3.34.57502: [udp sum ok] 38125 NXDomain q: AAAA? fnordahl-bastion.openstack.partnercloud1.lan. 0/1/0 ns: . SOA a.root-servers.net. nstld.verisign-grs.com. 2022033100 1800 900 604800 86400 (137)
Enabling distributed floating IP (aka DVR) the problem goes away.
When disabling hardware offload completely the problem goes away.
Running ovn-northd without [0] also makes the problem go away.
+ Versions:
+ OVS 2.17.0
+ OVN 22.03.0
+ Kernel 5.15.0-23-generic
+
+ Hardware: ConnectX-6 Dx MCX623106AN-CDAT
+ Firmware: 22.32.2004
+ Driver: OFED 5.5-1.0.3.2
+
0: https://github.com/ovn-
org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
** Description changed:
When hardware offload is enabled, CMS configures gateway routers and
centralized NAT, the source address of packets flowing through the
gateway are mangled after the connection preamble, not allowing the
connection to form. Return traffic on connections established from the
instance are also affected, even for UDP (DNS).
Example outside-in TCP connection attempt:
tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet), capture size 262144 bytes
05:52:04.286472 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 8865, offset 0, flags [DF], proto TCP (6), length 60)
10.11.2.11.57704 > 10.42.3.34.22: Flags [S], cksum 0x1e2a (correct), seq 3991821787, win 64240, options [mss 1460,sackOK,TS val 943370240 ecr 0,nop,wscale 7], length 0
05:52:04.286661 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.42.3.34.22 > 10.11.2.11.57704: Flags [S.], cksum 0x1990 (incorrect -> 0xdac4), seq 2903091535, ack 3991821788, win 62230, options [mss 8902,sackOK,TS val 2422032229 ecr 943370240,nop,wscale 7], length 0
05:52:04.287314 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 62, id 8866, offset 0, flags [DF], proto TCP (6), length 52)
10.11.2.11.57704 > 10.42.3.34.22: Flags [.], cksum 0x17c3 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 943370241 ecr 2422032229], length 0
05:52:04.287589 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 107: (tos 0x0, ttl 62, id 8867, offset 0, flags [DF], proto TCP (6), length 93)
0.0.0.0.57704 > 10.42.3.34.22: Flags [P.], cksum 0x017f (correct), seq 3991821788:3991821829, ack 2903091536, win 502, options [nop,nop,TS val 943370242 ecr 2422032229], length 41
Return traffic on connection-less UDP streams are also affected:
05:51:56.441418 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 56915, offset 0, flags [DF], proto UDP (17), length 90)
10.42.3.34.57502 > 194.169.254.1.53: [bad udp cksum 0xce4e -> 0x88d1!] 38125+ AAAA? fnordahl-bastion.openstack.partnercloud1.lan. (62)
05:51:56.443387 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 179: (tos 0x0, ttl 63, id 64535, offset 0, flags [DF], proto UDP (17), length 165)
0.0.0.0.53 > 10.42.3.34.57502: [udp sum ok] 38125 NXDomain q: AAAA? fnordahl-bastion.openstack.partnercloud1.lan. 0/1/0 ns: . SOA a.root-servers.net. nstld.verisign-grs.com. 2022033100 1800 900 604800 86400 (137)
Enabling distributed floating IP (aka DVR) the problem goes away.
When disabling hardware offload completely the problem goes away.
- Running ovn-northd without [0] also makes the problem go away.
+ Running ovn-northd without [0] also makes the problem go away regardless
+ of configuration.
Versions:
OVS 2.17.0
OVN 22.03.0
Kernel 5.15.0-23-generic
Hardware: ConnectX-6 Dx MCX623106AN-CDAT
Firmware: 22.32.2004
Driver: OFED 5.5-1.0.3.2
0: https://github.com/ovn-
org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1966981
Title:
[hwol] centralized gateway NAT mangles source address
Status in ovn package in Ubuntu:
Triaged
Bug description:
When hardware offload is enabled, CMS configures gateway routers and
centralized NAT, the source address of packets flowing through the
gateway are mangled after the connection preamble, not allowing the
connection to form. Return traffic on connections established from the
instance are also affected, even for UDP (DNS).
Example outside-in TCP connection attempt:
tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet), capture size 262144 bytes
05:52:04.286472 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 8865, offset 0, flags [DF], proto TCP (6), length 60)
10.11.2.11.57704 > 10.42.3.34.22: Flags [S], cksum 0x1e2a (correct), seq 3991821787, win 64240, options [mss 1460,sackOK,TS val 943370240 ecr 0,nop,wscale 7], length 0
05:52:04.286661 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.42.3.34.22 > 10.11.2.11.57704: Flags [S.], cksum 0x1990 (incorrect -> 0xdac4), seq 2903091535, ack 3991821788, win 62230, options [mss 8902,sackOK,TS val 2422032229 ecr 943370240,nop,wscale 7], length 0
05:52:04.287314 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 62, id 8866, offset 0, flags [DF], proto TCP (6), length 52)
10.11.2.11.57704 > 10.42.3.34.22: Flags [.], cksum 0x17c3 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 943370241 ecr 2422032229], length 0
05:52:04.287589 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 107: (tos 0x0, ttl 62, id 8867, offset 0, flags [DF], proto TCP (6), length 93)
0.0.0.0.57704 > 10.42.3.34.22: Flags [P.], cksum 0x017f (correct), seq 3991821788:3991821829, ack 2903091536, win 502, options [nop,nop,TS val 943370242 ecr 2422032229], length 41
Return traffic on connection-less UDP streams are also affected:
05:51:56.441418 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 56915, offset 0, flags [DF], proto UDP (17), length 90)
10.42.3.34.57502 > 194.169.254.1.53: [bad udp cksum 0xce4e -> 0x88d1!] 38125+ AAAA? fnordahl-bastion.openstack.partnercloud1.lan. (62)
05:51:56.443387 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4 (0x0800), length 179: (tos 0x0, ttl 63, id 64535, offset 0, flags [DF], proto UDP (17), length 165)
0.0.0.0.53 > 10.42.3.34.57502: [udp sum ok] 38125 NXDomain q: AAAA? fnordahl-bastion.openstack.partnercloud1.lan. 0/1/0 ns: . SOA a.root-servers.net. nstld.verisign-grs.com. 2022033100 1800 900 604800 86400 (137)
Enabling distributed floating IP (aka DVR) the problem goes away.
When disabling hardware offload completely the problem goes away.
Running ovn-northd without [0] also makes the problem go away
regardless of configuration.
Versions:
OVS 2.17.0
OVN 22.03.0
Kernel 5.15.0-23-generic
Hardware: ConnectX-6 Dx MCX623106AN-CDAT
Firmware: 22.32.2004
Driver: OFED 5.5-1.0.3.2
0: https://github.com/ovn-
org/ovn/commit/4deac4509abbedd6ffaecf27eed01ddefccea40a
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1966981/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list