[Bug 1994965] Re: [hwol] HW offload for traffic on direct ports without security group is broken the moment a direct port with stateful security group is created
Itai Levy
1994965 at bugs.launchpad.net
Thu Oct 27 12:22:30 UTC 2022
Note - this is not happening when a new direct port is created with
stateless SG
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1994965
Title:
[hwol] HW offload for traffic on direct ports without security group
is broken the moment a direct port with stateful security group is
created
Status in ovn package in Ubuntu:
New
Bug description:
Platform: Canonical Charmed OpenStack ("Yoga" release)
OS: 22.04 (Jammy), Kernel 5.15.0-48-generic, inbox drivers, no MOFED
OVS 2.17.2
OVN 22.03.0-0ubuntu1
NIC CX6Dx, LAG (LACP)
Issue description:
It is expected that when using direct ports without security group
(SG), conntrack will not be used and none of the OVS flows should
include a CT action. HW offload without CT action is very straight
forward and should work easily. This is indeed the result when running
such a test, however the moment a new direct port is created and
assigned with a stateful SG (even without allocating it to any VM),
the flows of the non-SG direct ports will start appearing as OVS flows
with CT actions.
As CT-based OVS flows are more complex to HW offload, this might
result in HW offload stop working for some running workloads the
moment a direct port with SG is created on the cloud.
Reproduction steps:
1. create geneve overlay management and data networks
2. create normal ports on the mgmt network with stateful security
group
3. Create direct ports with HW offload capability without security
group
openstack port create direct_overlay_no_sg1 --vnic-type=direct --network gen_data --binding-profile '{"capabilities":["switchdev"]}' --no-security-group --disable-port-security
openstack port create direct_overlay_no_sg2 --vnic-type=direct --network gen_data --binding-profile '{"capabilities":["switchdev"]}' --no-security-group --disable-port-security
4. Create instances with mgmt and data ports, and run traffic. The
example below is for OVS flows created when running iperf traffic
between the instances, all traffic is offloaded - keep the traffic
running
ufid:4bccc06d-5f3d-4167-acac-0d917349ce8f,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0),dp_hash(0/0),in_port(eth3),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:39:73:ee,dst=fa:16:3e:e0:44:fc),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=33.33.33.8/255.255.255.248,proto=6,tos=0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),
packets:56346204, bytes:502403395648, used:0.500s, offloaded:yes,
dp:tc,
actions:set(tunnel(tun_id=0x1,dst=172.16.0.2,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x50006}),flags(csum|key))),genev_sys_6081
ufid:8c14360c-6472-4939-976c-1a59787e6f42, skb_priority(0/0),tunnel(tun_id=0x1,src=172.16.0.2,dst=172.16.0.1,ttl=0/0,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x60005/0x7fffffff}),flags(+key)),skb_mark(0/0),ct_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:e0:44:fc,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:2086807, bytes:137729290, used:0.500s, offloaded:yes, dp:tc, actions:eth3
5. Create a new direct port with HW offload capability on the data
network with stateful security group, dont assign it to any instance
openstack security group create data_policy
openstack security group rule create data_policy --protocol icmp --ingress
openstack security group rule create data_policy --protocol icmp --egress
openstack security group rule create data_policy --protocol tcp --ingress --dst-port 5001:5200
openstack security group rule create data_policy --protocol udp --ingress --dst-port 4000:6000
openstack port create direct_overlay_sg1 --vnic-type=direct --network
gen_data --binding-profile '{"capabilities":["switchdev"]}'
--security-group data_policy
6. Check the OVS flows of the running iperf, they are now appear with
CT action and passing via CT chains
ufid:4bccc06d-5f3d-4167-acac-0d917349ce8f,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0),dp_hash(0/0),in_port(eth3),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:39:73:ee,dst=fa:16:3e:e0:44:fc),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=33.33.33.8/255.255.255.248,proto=6,tos=0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),
packets:64084134, bytes:571501263676, used:0.780s, offloaded:yes,
dp:tc, actions:ct(zone=6),recirc(0xa02)
ufid:28ffcced-c365-49e4-95c3-76e2fe5b6f5c,
skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xa02),dp_hash(0/0),in_port(eth3),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:39:73:ee,dst=fa:16:3e:e0:44:fc),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=33.33.33.8/255.255.255.248,proto=6,tos=0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),
packets:64083811, bytes:571498377162, used:0.780s, offloaded:yes,
dp:tc,
actions:set(tunnel(tun_id=0x1,dst=172.16.0.2,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x50006}),flags(csum|key))),genev_sys_6081
ufid:8c14360c-6472-4939-976c-1a59787e6f42,
skb_priority(0/0),tunnel(tun_id=0x1,src=172.16.0.2,dst=172.16.0.1,ttl=0/0,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x60005/0x7fffffff}),flags(+key)),skb_mark(0/0),ct_state(0/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:e0:44:fc,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0),
packets:6438796, bytes:424964356, used:1.190s, offloaded:yes, dp:tc,
actions:ct(zone=6),recirc(0xa03)
ufid:1898447a-0374-4fe9-b84a-770e597cbbe1,
skb_priority(0/0),tunnel(tun_id=0x1,src=172.16.0.2,dst=172.16.0.1,ttl=0/0,tp_dst=6081,geneve({class=0x102/0,type=0x80/0,len=4,0x60005/0}),flags(+key)),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xa03),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no),
packets:6441997, bytes:425176082, used:1.190s, offloaded:yes, dp:tc,
actions:eth3
7. Delete the direct port with the stateful security group, flows are
back to normal
8. Repeat the same test with RDMA traffic, the creation of the port
will severliy impact the running workload as some of the newly
created CT flows are not offloaded anymore to HW
ovs-appctl dpctl/dump-flows -m | grep "eth3"
ufid:ffbe4da4-5e02-4601-826d-dedcd22dacd4, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(eth3),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:39:73:ee,dst=fa:16:3e:e0:44:fc),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=32768/0x8000,dst=0/0), packets:11126607, bytes:46075194692, used:0.000s, offloaded:yes, dp:tc, actions:ct(zone=6),recirc(0xa11)
ufid:23b21267-7834-4194-b30b-801d623d4659,
skb_priority(0/0),skb_mark(0/0),ct_state(0x21/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xa11),dp_hash(0/0),in_port(eth3),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:39:73:ee,dst=fa:16:3e:e0:44:fc),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=33.33.33.8/255.255.255.248,proto=17,tos=0x2,ttl=0/0,frag=no),udp(src=0/0,dst=4096/0xf000),
packets:11126613, bytes:46075219532, used:0.000s, dp:tc,
actions:ct(commit,zone=6,label=0/0x1,nat(src)),set(tunnel(tun_id=0x1,dst=172.16.0.2,tos=0x2,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x50006}),flags(csum|key))),genev_sys_6081
ufid:40d741cd-d22d-456c-9c85-0f403ea79bd6,
skb_priority(0/0),tunnel(tun_id=0x1,src=172.16.0.2,dst=172.16.0.1,tos=0x2,ttl=0/0,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x60005/0x7fffffff}),flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:e0:44:fc,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=32768/0x8000,dst=4096/0x1000),
packets:1051226, bytes:50460024, used:10.160s, offloaded:yes, dp:tc,
actions:ct(zone=6),recirc(0xa12
ufid:3075008c-6cab-4dc0-8563-2a26983259e6,
skb_priority(0/0),tunnel(tun_id=0x1,src=172.16.0.2,dst=172.16.0.1,tos=0x2,ttl=0/0,tp_dst=6081,geneve({class=0x102/0,type=0x80/0,len=4,0x60005/0}),flags(+key)),skb_mark(0/0),ct_state(0x21/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xa12),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:e0:44:fc,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=17,tos=0/0,ttl=0/0,frag=no),udp(src=32768/0x8000,dst=0/0),
packets:1051226, bytes:50460024, used:10.160s, dp:tc,
actions:ct(commit,zone=6,label=0/0x1,nat(src)),eth3
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1994965/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list