Dropped packets with bonded interface

Mon Nov 14 10:56:20 UTC 2011

On 12.11.2011 11:07, Albert Chin wrote:
> On Sat, Nov 12, 2011 at 01:49:40PM +1100, James Gray wrote:
>> On 12/11/2011, at 3:58 AM, Albert Chin wrote:
>>
>>> I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR
>>> motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting
>>> dropped packets on a bonded interface:
>>>
>>> $ ifconfig -a
>>> ...
>>> bond0     Link encap:Ethernet  HWaddr 00:1b:21:b7:21:ea  
>>>          BROADCAST PROMISC MASTER MULTICAST  MTU:1500  Metric:1
>>>          RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0
>>>          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
>>>          collisions:0 txqueuelen:0 
>>>          RX bytes:306528 (306.5 KB)  TX bytes:272 (272.0 B)
>>
>> --- >8 --- Snipped
>>
>>> $ cat /etc/network/interfaces
>>> ...
>>> auto bond0
>>> iface bond0 inet manual
>>>  bond-slaves none
>>>  bond_mode 802.3ad
>>>  bond_miimon 100
>>>
>>> auto eth2
>>> iface eth2 inet manual
>>>  bond-master bond0
>>>
>>> auto eth3
>>> iface eth3 inet manual
>>>  bond-master bond0
>>>
>>> 2. Why isn't the bond0 interface "up"?
>>
>> The dropped packets on bond0 are probably control packets the kernel
>> is ignoring because you haven't enslaved any interfaces.  Which
>> leads to question 2.  Try modifying the "iface bond0" stanza in
>> /etc/network/interfaces:
>>
>> auto bond0
>> iface bond0 inet manual
>>   bond-slaves eth2 eth3   # No slaves == nothing for bond0 to bind to!
>>   bond_mode 802.3ad
>>   bond_miimon 100
>>
>> You might want to add an "updelay" in there too if your interfaces
>> take a little while to synchronise with the switch (easy for the
>> switch to block the port until spanning tree has done its thing
>> unless you've specify "portfast" on the switch port).  Then bounce
>> the network and see what happens.  When it's all running properly,
>> you should see something like this:
> 
> Thanks. I tried that but it didn't work. I updated bond0 in
> /etc/network/interfaces to:
>   auto bond0
>   iface bond0 inet static
>     address 10.191.62.2
>     netmask 255.255.255.0
>     broadcast 10.191.62.255
>     bond-slaves eth2 eth3
>     bond-primary eth2 eth3
>     bond-mode 802.3ad
>     bond-lacp_rate fast
>     bond-miimon 100
> 
> When I boot with this, I get:
>   # ifconfig -a
>   bond0     Link encap:Ethernet  HWaddr 00:1b:21:b7:21:ea  
>             inet addr:10.191.62.2  Bcast:10.191.62.255 Mask:255.255.255.0
>             inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link
>             UP BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
>             RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>             collisions:0 txqueuelen:0 
>             RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
>   eth2      Link encap:Ethernet  HWaddr 00:1b:21:b7:21:ea  
>             UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>             RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>             collisions:0 txqueuelen:1000 
>             RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>             Memory:b2420000-b2440000 
> 
>   eth3      Link encap:Ethernet  HWaddr 00:1b:21:b7:21:ea  
>             UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>             RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>             collisions:0 txqueuelen:1000 
>             RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>             Memory:b2400000-b2420000 
> 
>   # cat /proc/net/bonding/bond0                      
>   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
> 
>   Bonding Mode: IEEE 802.3ad Dynamic link aggregation
>   Transmit Hash Policy: layer2 (0)
>   MII Status: down
>   MII Polling Interval (ms): 100
>   Up Delay (ms): 0
>   Down Delay (ms): 0
> 
>   802.3ad info
>   LACP rate: fast
>   Aggregator selection policy (ad_select): stable
>   bond bond0 has no active aggregator
> 
>   Slave Interface: eth2
>   MII Status: up
>   Speed: 1000 Mbps
>   Duplex: full
>   Link Failure Count: 1
>   Permanent HW addr: 00:1b:21:b7:21:ea
>   Aggregator ID: N/A
>   Slave queue ID: 0
> 
>   Slave Interface: eth3
>   MII Status: up
>   Speed: 1000 Mbps
>   Duplex: full
>   Link Failure Count: 1
>   Permanent HW addr: 00:1b:21:b7:21:eb
>   Aggregator ID: N/A
>   Slave queue ID: 0
> 
> What's odd is that after the system comes up with the above config, if
> I do the following:
>   # ip link set dev bond0 up 
>   # ifenslave bond0 eth2 eth3
>   # ifconfig bond0 10.191.62.2 netmask 255.255.255.0
>   # ifconfig bond0
>   bond0     Link encap:Ethernet  HWaddr 00:1b:21:b7:21:ea  
>             inet addr:10.191.62.2  Bcast:10.191.62.255 Mask:255.255.255.0
>             inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link
>             UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>             RX packets:17 errors:0 dropped:17 overruns:0 frame:0
>             TX packets:27 errors:0 dropped:0 overruns:0 carrier:0
>             collisions:0 txqueuelen:0 
>             RX bytes:2108 (2.1 KB)  TX bytes:3126 (3.1 KB)
> 
>   # cat /proc/net/bonding/bond0
>   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
> 
>   Bonding Mode: IEEE 802.3ad Dynamic link aggregation
>   Transmit Hash Policy: layer2 (0)
>   MII Status: up
>   MII Polling Interval (ms): 100
>   Up Delay (ms): 0
>   Down Delay (ms): 0
> 
>   802.3ad info
>   LACP rate: fast
>   Aggregator selection policy (ad_select): stable
>   Active Aggregator Info:
>           Aggregator ID: 1
>           Number of ports: 2
>           Actor Key: 17
>           Partner Key: 24
>           Partner Mac Address: 00:04:96:18:54:d5
> 
>   Slave Interface: eth2
>   MII Status: up
>   Speed: 1000 Mbps
>   Duplex: full
>   Link Failure Count: 0
>   Permanent HW addr: 00:1b:21:b7:21:ea
>   Aggregator ID: 1
>   Slave queue ID: 0
> 
>   Slave Interface: eth3
>   MII Status: up
>   Speed: 1000 Mbps
>   Duplex: full
>   Link Failure Count: 0
>   Permanent HW addr: 00:1b:21:b7:21:eb
>   Aggregator ID: 1
>   Slave queue ID: 0
> 
> So it seems something odd is going on. Notice that I continue to get
> dropped packets after this "working" config.
> 
>> Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
>>
>> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
>> Transmit Hash Policy: layer2 (0)
>> MII Status: up
>> MII Polling Interval (ms): 100
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>>
>> 802.3ad info
>> LACP rate: slow
>> Aggregator selection policy (ad_select): stable
>> Active Aggregator Info:
>> 	Aggregator ID: 1
>> 	Number of ports: 2
>> 	Actor Key: 17
>> 	Partner Key: 53
>> 	Partner Mac Address: 74:ea:3a:ba:35:e4
>>
>> Slave Interface: eth0
>> MII Status: up
>> Link Failure Count: 0
>> Permanent HW addr: 00:08:9b:c4:a6:f4
>> Aggregator ID: 1
>>
>> Slave Interface: eth1
>> MII Status: up
>> Link Failure Count: 0
>> Permanent HW addr: 00:08:9b:c4:a6:f5
>> Aggregator ID: 1
>>
>>
>> As always, there's some documentation to read:
>> https://help.ubuntu.com/community/UbuntuBonding
> 
> Your output above looks different than the "Check operation" section
> of https://help.ubuntu.com/community/UbuntuBonding.
> 
My best guess about the dropped packages would be some that arrive before being
set up... (like broadcasts)
Just as a note that the recommendation for the bond-slaves option is none. This
is because the bond interface should get the slaves added by the slave
interface. Probably requiring auto lines for the slaves, too (the README and
examples coming with the package should reflect this by now).
The interface did not come up before because there was not network configuration
set for the bond interface.

-Stefan