Network Bonding issue 12.04.2LTS

Mon Aug 19 07:07:09 UTC 2013

Hello All,

I'm having issues with our new Dell R515/C6145's and Ubuntu 12.04 LTS 

Installed with the mini.iso from http://archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/mini.iso
The R515 specs are
32Gb ram
AMD Hex core CPU 2.6Ghz
Dual port Intel 10 Gbit SFP+ Cards

The C6156 specs
192Gb RAM
4x AMD 16c CPU 
Dual port Intel 10gbit SPF+ Cards

My issues seem to be with bonding drivers and kernel upgrades.

If I attempted to upgrade the kernel to raring via

apt-get install linux-image-generic-lts-raring

all goes well until the reboot and the network card receive an error -5 in dmesg and the interfaces don't come up.

So I've upgraded the kernel using http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.10.7-saucy/
Packages install ok, however now my console is blank upon reboot! (it's via the irmc lights out card)

I'm able to logon to the server ok via ssh etc. so I did some testing again,

First with bonding using LACP Layer 3+4 Hashing combined throughput TX performance is always around 10gbit however when receiving is the R515 is able to achieve 18gbps from the c6145.

C6154: iperf -s
R515: iperf -c c6154IP -P 2

I get around 9-10Gbps which is split evenly between eth0 & eth1

However When reversed (sent from the C6145's only)
I get around 18gbps which is also split evenly between the interfaces as previously mentioned.

More strangely 
on the C6145's, between the 2  C6145's the throughput will never exceed 10gbps (max speed of a single interface) even tho the traffic is evenly distributed across both interfaces.

So it seems to be something around bonding is causing the issue,  
When the nic's are not bonded I can get 10/10 on both nic's 
However when  using bonding it seems to be 1 way "rate limited" in the R515 case its "rate limited" outbound and on the C6145's its "rate limited" inbound.

I also ruled out the switch by directly connecting the servers to each other which I observed the same issues.

Very confusing... Any ideas?

Some Info Below from the R515's.
----------
Current kernel
Linux r515 3.10.7-031007-generic #201308150319 SMP Thu Aug 15 07:20:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
-----------------
/etc/network/interfaces (bond settings) and LACP info

auto bond0
iface bond0 inet manual
      bond_mode 802.3ad
      bond_miimon 100
      bond_lacp_rate 1
      bond_xmit_hash_policy layer3+4
      bond-slaves none
      post-up ifconfig bond0 mtu 9000
-----------------
cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 1
        Actor Key: 33
        Partner Key: 13
        Partner Mac Address: 00:01:e8:8b:b0:5d

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:e2:ba:46:e4:e8
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:e2:ba:46:e4:e9
Aggregator ID: 2
Slave queue ID: 0
---------------
01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
---------------
# dmesg | grep ixgbe
[    1.211896] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.13.10-k
[    1.211900] ixgbe: Copyright (c) 1999-2013 Intel Corporation.
[    1.366736] ixgbe 0000:01:00.0: irq 75 for MSI/MSI-X
[    1.366749] ixgbe 0000:01:00.0: irq 76 for MSI/MSI-X
[    1.366761] ixgbe 0000:01:00.0: irq 77 for MSI/MSI-X
[    1.366772] ixgbe 0000:01:00.0: irq 78 for MSI/MSI-X
[    1.366783] ixgbe 0000:01:00.0: irq 79 for MSI/MSI-X
[    1.366794] ixgbe 0000:01:00.0: irq 80 for MSI/MSI-X
[    1.366805] ixgbe 0000:01:00.0: irq 81 for MSI/MSI-X
[    1.366831] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 6, Tx Queue count = 6
[    1.366961] ixgbe 0000:01:00.0: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:46:e4:e8
[    1.367295] ixgbe 0000:01:00.0: MAC: 2, PHY: 9, SFP+: 3, PBA No: G18786-004
[    1.368975] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[    1.522597] ixgbe 0000:01:00.1: irq 82 for MSI/MSI-X
[    1.522621] ixgbe 0000:01:00.1: irq 83 for MSI/MSI-X
[    1.522633] ixgbe 0000:01:00.1: irq 84 for MSI/MSI-X
[    1.522651] ixgbe 0000:01:00.1: irq 85 for MSI/MSI-X
[    1.522671] ixgbe 0000:01:00.1: irq 86 for MSI/MSI-X
[    1.522683] ixgbe 0000:01:00.1: irq 87 for MSI/MSI-X
[    1.522704] ixgbe 0000:01:00.1: irq 88 for MSI/MSI-X
[    1.522729] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 6, Tx Queue count = 6
[    1.522853] ixgbe 0000:01:00.1: (PCI Express:5.0GT/s:Width x8) 90:e2:ba:46:e4:e9
[    1.523191] ixgbe 0000:01:00.1: MAC: 2, PHY: 9, SFP+: 4, PBA No: G18786-004
[    1.524879] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[    7.580548] ixgbe 0000:01:00.0: registered PHC device on eth0
[    7.713068] ixgbe 0000:01:00.1: registered PHC device on eth1
[    7.745940] ixgbe 0000:01:00.0 eth0: detected SFP+: 3
[    7.814361] ixgbe 0000:01:00.0 eth0: changing MTU from 1500 to 9000
[    7.877278] ixgbe 0000:01:00.1 eth1: detected SFP+: 4
[    8.016991] ixgbe 0000:01:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[    8.021318] ixgbe 0000:01:00.1 eth1: changing MTU from 1500 to 9000
[    8.081923] ixgbe 0000:01:00.0 eth0: detected SFP+: 3
[    8.300774] ixgbe 0000:01:00.1 eth1: detected SFP+: 4
[    8.324760] ixgbe 0000:01:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[    8.532644] ixgbe 0000:01:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
------------

Regards,
Quenten Grasso