asymetric GigE speeds between Linux and FreeBSD

security security at jim-liesl.org
Tue May 22 00:05:15 UTC 2007


I've cross posted this to the freebsd-net list. I'm new to the
(k)ubuntu lists, so if this isn't the right place to post, please let me
know.

Summary: Using iperf to measure TCP net speed between a linux (kubuntu
edgy) and freebsd box over gigE, I see significant speed difference
depending
on the data direction. Pushing data from the freebsd box to the linux box,
I average about 500Gb/s. Pushing data from the linux box to the freebsd
box, I see about 300Gb/s. It would seem that there is something
limiting either the transmit side of the linux box or the recv side of
the freebsd box. Any suggestions on how to narrow down who's tuning
needs more work would also be welcome. Just for grins, I tried upping
the txqueuelen on the linux box to 1500, but no help.

I'd like to get closer to the theoretical speed of this link (I'd be
happy if I could get 600-700 both ways), but the real puzzler is the
difference. The client is the linux box @192.168.1.104 and the iperf
server is the freebsd box @ 192.168.1.15 . Neither box is cpu or net
busy during testing. Both systems only have one nic. I played with
larger window sizes, but it only had a minor affect. I went with the
Intel Pro/1000's based on their good rep in both the linux and freebsd
communities.

Client connecting to 192.168.1.15, TCP port 5001
TCP window size:   128 KByte (default)
------------------------------------------------------------
[  6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001
[  6]  0.0-10.0 sec    369 MBytes    309 Mbits/sec
[  5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963
[  5]  0.0-10.0 sec    597 MBytes    500 Mbits/sec


Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are
Belkin 5e), both systems are on this switch.

FreeBSD box:
FreeBSD 6.1-RELEASE-p10
1.9Ghz Athlon / 1 gig of main mem
Abit/nforce2 MB+chipset (onboard nic disabled in bios)
Intel Pro/1000GT NIC
sysctl.conf:
kern.ipc.maxsockbuf=8192000
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
sysctl reports:
kern.ipc.nmbjumbo16: 0
kern.ipc.nmbjumbo9: 0
kern.ipc.nmbjumbop: 0
kern.ipc.nmbclusters: 25600
net.inet.tcp.rfc1323: 1

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1
        inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255
        ether 00:0e:0c:da:1a:3c
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

Linux box:
Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686
GNU/Linux
Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem.
I run  tcp_tune.sh at boot on the linux box
#!/bin/bash
echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem
echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem
echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem

/proc/sys/net/ipv4/tcp_window_scaling is 1
/proc/sys/net/ipv4/tcp_timestamps is 1

eth1      Link encap:Ethernet  HWaddr 00:0E:0C:DA:1A:3B
          inet addr:192.168.1.104  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:147638720 (140.7 MiB)  TX bytes:1109677958 (1.0 GiB)
          Base address:0xd800 Memory:e2020000-e2040000

Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
        Link detected: yes










More information about the kernel-team mailing list