[Bug 2047494] [NEW] Failure to Forward Traffic for VIP-bound Floating IP in HA Router
liujinxin
2047494 at bugs.launchpad.net
Wed Dec 27 06:58:20 UTC 2023
Public bug reported:
- OpenStack version: Based on /stable/victoria
Environment information: mgt04 and mgt05 are snat nodes for Neutron.
- Description:
vm1 is a virtual machine of router1 without a floating IP (fip). The network:router_centralized_snat port is on mgt05.
vm2 is a virtual machine of router2 with a VIP that is bound to fip1. Router2 is a high availability (HA) router with l3agents running on mgt04 and mgt05. Currently, mgt04 is the master node for this router.
- The traffic flow for vm1 pinging vm2 is as follows:
tap-vm1 ----> qrouter1-netns ----> snat-netns-router1-mgt05 (this netns has learned the ARP for fip1 and sets the destination MAC address of the packet to fip1's MAC address, sending it out through the qg-port for OVS flow table forwarding) ----> ofproto/trace reveals that the packet, based on the fip1's destination MAC, matches the following flow table and is sent to the qg-port of snat-netns-of-router2 on the local node, mgt05. However, in reality, since mgt05 is the standby node for qrouter2, the link status of the qg-port is down, and there are no rules for the fip bound to the VIP on the qg-port, resulting in the traffic not being forwarded.
[root at mgt05 ~]# ovs-appctl ofproto/trace br-int in_port=qg-43377cd4-5e
fa163e6ea663fa163e5d04e8080045000054d15740003e019f7264e400c864e4014f08008d604c334455d87b8a6500000000ac620c0000000000101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
Flow: icmp,in_port=266,vlan_tci=0x0000,dl_src=fa:16:3e:5d:04:e8,dl_dst=fa:16:
3e:6e:a6:63,nw_src=100.228.0.200,nw_dst=100.228.1.79,nw_tos=0,nw_ecn=0,nw_ttl=
62,icmp_type=8,icmp_code=0
bridge("br-int")
----------------
0. priority 0, cookie 0x3d6d07f41c6793a3
goto_table:60
60. in_port=266,dl_src=fa:16:3e:5d:04:e8, priority 9, cookie 0x3d6d07f41c6793a3
set_field:0x2->reg6
resubmit(,61)
61. reg6=0x2,dl_dst=fa:16:3e:6e:a6:63, priority 12, cookie 0x3d6d07f41c6793a3
output:143
Final flow: icmp,reg6=0x2,in_port=266,vlan_tci=0x0000,dl_src=fa:16:3e:5d:04:e8
,dl_dst=fa:16:3e:6e:a6:63,nw_src=100.228.0.200,nw_dst=100.228.1.79,nw_tos=0,
nw_ecn=0,nw_ttl=62,icmp_type=8,icmp_code=0
Megaflow: recirc_id=0,eth,ip,in_port=266,dl_src=fa:16:3e:5d:04:e8,dl_dst=fa:16
:3e:6e:a6:63,nw_frag=no
Datapath actions: 33
[root at mgt05 ~]#
[root at mgt05 ~]# ovs-dpctl show |grep 33
port 33: qg-111ff616-09 (internal)
[root at mgt05 ~]# ip netns exec snat-ed122990-2d34-4d9f-9fed-189ba0243d06 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
266: ha-931f41ef-99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:52:ff:e7 brd ff:ff:ff:ff:ff:ff
inet 169.254.192.54/18 brd 169.254.255.255 scope global ha-931f41ef-99
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe52:ffe7/64 scope link
valid_lft forever preferred_lft forever
268: sg-0814de55-6a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:4e:19:b6 brd ff:ff:ff:ff:ff:ff
272: qg-111ff616-09: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether fa:16:3e:6e:a6:63 brd ff:ff:ff:ff:ff:ff
# openstack port show 111ff616-09b0-46ec-b46d-078d3050e62f
+-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | mgt04 |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | |
| device_id | ed122990-2d34-4d9f-9fed-189ba0243d06 |
| device_owner | network:router_gateway
** Affects: neutron (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/2047494
Title:
Failure to Forward Traffic for VIP-bound Floating IP in HA Router
Status in neutron package in Ubuntu:
New
Bug description:
- OpenStack version: Based on /stable/victoria
Environment information: mgt04 and mgt05 are snat nodes for Neutron.
- Description:
vm1 is a virtual machine of router1 without a floating IP (fip). The network:router_centralized_snat port is on mgt05.
vm2 is a virtual machine of router2 with a VIP that is bound to fip1. Router2 is a high availability (HA) router with l3agents running on mgt04 and mgt05. Currently, mgt04 is the master node for this router.
- The traffic flow for vm1 pinging vm2 is as follows:
tap-vm1 ----> qrouter1-netns ----> snat-netns-router1-mgt05 (this netns has learned the ARP for fip1 and sets the destination MAC address of the packet to fip1's MAC address, sending it out through the qg-port for OVS flow table forwarding) ----> ofproto/trace reveals that the packet, based on the fip1's destination MAC, matches the following flow table and is sent to the qg-port of snat-netns-of-router2 on the local node, mgt05. However, in reality, since mgt05 is the standby node for qrouter2, the link status of the qg-port is down, and there are no rules for the fip bound to the VIP on the qg-port, resulting in the traffic not being forwarded.
[root at mgt05 ~]# ovs-appctl ofproto/trace br-int in_port=qg-43377cd4-5e
fa163e6ea663fa163e5d04e8080045000054d15740003e019f7264e400c864e4014f08008d604c334455d87b8a6500000000ac620c0000000000101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
Flow: icmp,in_port=266,vlan_tci=0x0000,dl_src=fa:16:3e:5d:04:e8,dl_dst=fa:16:
3e:6e:a6:63,nw_src=100.228.0.200,nw_dst=100.228.1.79,nw_tos=0,nw_ecn=0,nw_ttl=
62,icmp_type=8,icmp_code=0
bridge("br-int")
----------------
0. priority 0, cookie 0x3d6d07f41c6793a3
goto_table:60
60. in_port=266,dl_src=fa:16:3e:5d:04:e8, priority 9, cookie 0x3d6d07f41c6793a3
set_field:0x2->reg6
resubmit(,61)
61. reg6=0x2,dl_dst=fa:16:3e:6e:a6:63, priority 12, cookie 0x3d6d07f41c6793a3
output:143
Final flow: icmp,reg6=0x2,in_port=266,vlan_tci=0x0000,dl_src=fa:16:3e:5d:04:e8
,dl_dst=fa:16:3e:6e:a6:63,nw_src=100.228.0.200,nw_dst=100.228.1.79,nw_tos=0,
nw_ecn=0,nw_ttl=62,icmp_type=8,icmp_code=0
Megaflow: recirc_id=0,eth,ip,in_port=266,dl_src=fa:16:3e:5d:04:e8,dl_dst=fa:16
:3e:6e:a6:63,nw_frag=no
Datapath actions: 33
[root at mgt05 ~]#
[root at mgt05 ~]# ovs-dpctl show |grep 33
port 33: qg-111ff616-09 (internal)
[root at mgt05 ~]# ip netns exec snat-ed122990-2d34-4d9f-9fed-189ba0243d06 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
266: ha-931f41ef-99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:52:ff:e7 brd ff:ff:ff:ff:ff:ff
inet 169.254.192.54/18 brd 169.254.255.255 scope global ha-931f41ef-99
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe52:ffe7/64 scope link
valid_lft forever preferred_lft forever
268: sg-0814de55-6a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:4e:19:b6 brd ff:ff:ff:ff:ff:ff
272: qg-111ff616-09: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether fa:16:3e:6e:a6:63 brd ff:ff:ff:ff:ff:ff
# openstack port show 111ff616-09b0-46ec-b46d-078d3050e62f
+-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | mgt04 |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | |
| device_id | ed122990-2d34-4d9f-9fed-189ba0243d06 |
| device_owner | network:router_gateway
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/2047494/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list