[Bug 1629053] [NEW] ovs bridge breaking TCP between two virtio net devices when checksum offload on

bugproxy bugproxy at us.ibm.com
Thu Sep 29 18:09:59 UTC 2016


Public bug reported:

== Comment: #0 - Gowrishankar Muthukrishnan - 2016-09-29 13:06:10 ==
---Problem Description---

Two kvm guests, each booted with a vnic (virtio_net) in ovs bridge in host seem 
to fail in establishing TCP when virtio turned on for tcp checksum offload.
Where as, these guests are able to complete TCP when vnics are over linux
bridge in host.

tcpdump on destination guest that confirms SYN retransmission, in OVS
case:

12:21:56.588835 52:54:c1:a2:7c:63 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 49483, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.126.3.51944 > 10.0.126.2.22: Flags [S], cksum 0x1034 (incorrect -> 0x0482), seq 621922575, win 29200, options [mss 1460,sackOK,TS val 329796 ecr 0,nop,wscale 7], length 0
12:21:57.588220 52:54:c1:a2:7c:63 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 49484, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.126.3.51944 > 10.0.126.2.22: Flags [S], cksum 0x1034 (incorrect -> 0x0388), seq 621922575, win 29200, options [mss 1460,sackOK,TS val 330046 ecr 0,nop,wscale 7], length 0

tcpdump on destination guest that confirms SYN/SYN-ACK/ACK , in bridge
case:

12:34:53.140513 52:54:c1:a2:7c:64 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 57464, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.126.3.41722 > 10.1.126.2.22: Flags [S], cksum 0x1036 (incorrect -> 0xb8a4), seq 1286151909, win 29200, options [mss 1460,sackOK,TS val 523933 ecr 0,nop,wscale 7], length 0
12:34:53.140729 52:54:c1:a2:7c:54 > 52:54:c1:a2:7c:64, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.126.2.22 > 10.1.126.3.41722: Flags [S.], cksum 0x1036 (incorrect -> 0xe19a), seq 3214677586, ack 1286151910, win 28960, options [mss 1460,sackOK,TS val 524786 ecr 523933,nop,wscale 7], length 0
12:34:53.141253 52:54:c1:a2:7c:64 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 57465, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.126.3.41722 > 10.1.126.2.22: Flags [.], cksum 0x102e (incorrect -> 0x80a1), ack 1, win 229, options [nop,nop,TS val 523934 ecr 524786], length 0
 
---uname output---
Linux ip9-114-219-126gate 4.4.0-9136-generic #55-Ubuntu SMP Fri Aug 26 05:56:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
 
Machine Type = P8 LE 
  
---Steps to Reproduce---
 1. Create a ovs bridge "ovsbr0"
2. Create a kvm guest (preferably same as host - Ubuntu 16.10) carrying a vnic
   (as shown below) in ovsbr0
   
   <interface type='bridge'>
      <mac address='52:54:c1:a2:7c:53'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'/>
      <model type='virtio'/>
   </interface>

   Similarly create another kvm guest but with vnic of different mac address.
   
3. Assign a static IP over vnic inside each guest.
   For eg, if enp0s3 is vnic:
   sudo ifconfig enp0s3 10.0.126.2/24 up (in guest 1)
   sudo ifconfig enp0s3 10.0.126.3/24 up (in guest 2)
   
4. Ping and ensure ARP completes.
   ping 10.0.126.3 (from guest 2)
   
5. Check tcp scum offload ON as by default.
   ethtool -k enp0s3
   
   Features for enp0s3:
    rx-checksumming: on [fixed]
    tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]

6. Check if TCP establishes between two guests on above IPs, using nc (netcat).
   nc 10.0.126.2 22
   
   As you would not see "SSH-2.0-OpenSSH_7.3p1 Ubuntu-1" as response, this 
   confirms TCP over port did not establish.
   
   You may run tcpdump on each port while running nc to confirm invalid 
   checksum received in tcp (as shown in problem description).

** Affects: openvswitch (Ubuntu)
     Importance: Undecided
     Assignee: Taco Screen team (taco-screen-team)
         Status: New


** Tags: architecture-ppc64le bugnameltc-147080 severity-medium targetmilestone-inin---

** Tags added: architecture-ppc64le bugnameltc-147080 severity-medium
targetmilestone-inin---

** Changed in: ubuntu
     Assignee: (unassigned) => Taco Screen team (taco-screen-team)

** Package changed: ubuntu => openvswitch (Ubuntu)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1629053

Title:
  ovs bridge breaking TCP between two virtio net devices when checksum
  offload on

Status in openvswitch package in Ubuntu:
  New

Bug description:
  == Comment: #0 - Gowrishankar Muthukrishnan - 2016-09-29 13:06:10 ==
  ---Problem Description---

  Two kvm guests, each booted with a vnic (virtio_net) in ovs bridge in host seem 
  to fail in establishing TCP when virtio turned on for tcp checksum offload.
  Where as, these guests are able to complete TCP when vnics are over linux
  bridge in host.

  tcpdump on destination guest that confirms SYN retransmission, in OVS
  case:

  12:21:56.588835 52:54:c1:a2:7c:63 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 49483, offset 0, flags [DF], proto TCP (6), length 60)
      10.0.126.3.51944 > 10.0.126.2.22: Flags [S], cksum 0x1034 (incorrect -> 0x0482), seq 621922575, win 29200, options [mss 1460,sackOK,TS val 329796 ecr 0,nop,wscale 7], length 0
  12:21:57.588220 52:54:c1:a2:7c:63 > 52:54:c1:a2:7c:53, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 49484, offset 0, flags [DF], proto TCP (6), length 60)
      10.0.126.3.51944 > 10.0.126.2.22: Flags [S], cksum 0x1034 (incorrect -> 0x0388), seq 621922575, win 29200, options [mss 1460,sackOK,TS val 330046 ecr 0,nop,wscale 7], length 0

  tcpdump on destination guest that confirms SYN/SYN-ACK/ACK , in bridge
  case:

  12:34:53.140513 52:54:c1:a2:7c:64 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 57464, offset 0, flags [DF], proto TCP (6), length 60)
      10.1.126.3.41722 > 10.1.126.2.22: Flags [S], cksum 0x1036 (incorrect -> 0xb8a4), seq 1286151909, win 29200, options [mss 1460,sackOK,TS val 523933 ecr 0,nop,wscale 7], length 0
  12:34:53.140729 52:54:c1:a2:7c:54 > 52:54:c1:a2:7c:64, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
      10.1.126.2.22 > 10.1.126.3.41722: Flags [S.], cksum 0x1036 (incorrect -> 0xe19a), seq 3214677586, ack 1286151910, win 28960, options [mss 1460,sackOK,TS val 524786 ecr 523933,nop,wscale 7], length 0
  12:34:53.141253 52:54:c1:a2:7c:64 > 52:54:c1:a2:7c:54, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 57465, offset 0, flags [DF], proto TCP (6), length 52)
      10.1.126.3.41722 > 10.1.126.2.22: Flags [.], cksum 0x102e (incorrect -> 0x80a1), ack 1, win 229, options [nop,nop,TS val 523934 ecr 524786], length 0
   
  ---uname output---
  Linux ip9-114-219-126gate 4.4.0-9136-generic #55-Ubuntu SMP Fri Aug 26 05:56:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
   
  Machine Type = P8 LE 
    
  ---Steps to Reproduce---
   1. Create a ovs bridge "ovsbr0"
  2. Create a kvm guest (preferably same as host - Ubuntu 16.10) carrying a vnic
     (as shown below) in ovsbr0
     
     <interface type='bridge'>
        <mac address='52:54:c1:a2:7c:53'/>
        <source bridge='ovsbr0'/>
        <virtualport type='openvswitch'/>
        <model type='virtio'/>
     </interface>

     Similarly create another kvm guest but with vnic of different mac address.
     
  3. Assign a static IP over vnic inside each guest.
     For eg, if enp0s3 is vnic:
     sudo ifconfig enp0s3 10.0.126.2/24 up (in guest 1)
     sudo ifconfig enp0s3 10.0.126.3/24 up (in guest 2)
     
  4. Ping and ensure ARP completes.
     ping 10.0.126.3 (from guest 2)
     
  5. Check tcp scum offload ON as by default.
     ethtool -k enp0s3
     
     Features for enp0s3:
      rx-checksumming: on [fixed]
      tx-checksumming: on
  	tx-checksum-ipv4: off [fixed]
  	tx-checksum-ip-generic: on
  	tx-checksum-ipv6: off [fixed]
  	tx-checksum-fcoe-crc: off [fixed]
  	tx-checksum-sctp: off [fixed]

  6. Check if TCP establishes between two guests on above IPs, using nc (netcat).
     nc 10.0.126.2 22
     
     As you would not see "SSH-2.0-OpenSSH_7.3p1 Ubuntu-1" as response, this 
     confirms TCP over port did not establish.
     
     You may run tcpdump on each port while running nc to confirm invalid 
     checksum received in tcp (as shown in problem description).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1629053/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list