[Bug 1722584] Re: [SRU] Return traffic from metadata service may get dropped by hypervisor due to wrong checksum

Edward Hope-Morley edward.hope-morley at canonical.com
Thu Jul 11 11:24:38 UTC 2019


Xenial+Stein verified using [Test Case]

Test output:

root at juju-92f0c2-lp1722584-sru-5:~# dpkg -l| grep neutron-l3-agent
ii neutron-l3-agent 2:14.0.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
root at juju-92f0c2-lp1722584-sru-5:~# sudo ip netns exec qrouter-32ea60b2-ac9b-4a16-8933-63818eb71568 iptables -t mangle -S| grep 9697
root at juju-92f0c2-lp1722584-sru-5:~#


** Tags removed: verification-rocky-done verification-stein-needed
** Tags added: verification-rocky-needed verification-stein-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1722584

Title:
  [SRU] Return traffic from metadata service may get dropped by
  hypervisor due to wrong checksum

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Committed
Status in neutron source package in Cosmic:
  Fix Committed
Status in neutron source package in Disco:
  Fix Released
Status in neutron source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Prior addition of code to add checksum rules was found to cause problems with newer kernels. Patch subsequently reverted so this request is to backport those patches to the ubuntu archives.

  [Test Case]
  * deploy openstack (>= queens)
  * create router/network/instance (dvr=false,l3ha=false)
  * go to router ns on neutron-gateway and check that the following returns nothing
  sudo ip netns exec qrouter-<id> iptables -t mangle -S| grep '\--sport 9697 -j CHECKSUM --checksum-fill'

  [Regression Potential]
  Backporting the revert patch will mean that routers created with this patch will no longer have a checksum rule added for metadata tcp packets. The original patch added a rule that turned out not to be the fix for the root issue and was subsequently found to cause problems with kernels < 4.19 since it was never intended for gso tcp packets to have their checksum verified using this type of rule. So, removal of this rule (by addition of the revert patch) is not intended to change behaviour at all. The only potential side-effect is that rules that were already created will not be cleaned up (until node reboot or router recreate) and in an L3HA config you could end up with some router instances having the rule and some not depending on whether they were created before or after the patch was included.

  [Other Info]
  This revert patch does not remove rules added by the original patch so manual cleanup of those old rules is required.

  -----------------------------------------------------------------------------
  We have a problem with the metadata service not being responsive, when the proxied in the router namespace on some of our networking nodes after upgrading to Ocata (Running on CentOS 7.4, with the RDO packages).

  Instance routes traffic to 169.254.169.254 to it's default gateway.
  Default gateway is an OpenStack router in a namespace on a networking node.

  - Traffic gets sent from the guest,
  - to the router,
  - iptables routes it to the metadata proxy service,
  - response packet gets routed back, leaving the namespace
  - Hypervisor gets the packet in
  - Checksum of packet is wrong, and the packet gets dropped before putting it on the bridge

  Based on the following bug https://bugs.launchpad.net/openstack-
  ansible/+bug/1483603, we found that adding the following iptable rule
  in the router namespace made this work again: 'iptables -t mangle -I
  POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill'

  (NOTE: The rule from the 1st comment to the bug did solve access to
  the metadata service, but the lack of precision introduced other
  problems with the network)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1722584/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list