Note: An update as been added below
I'm having a very strange issue with a Linux server (Debian 6.0.6
) using 802.3ad
NIC bonding. Occasionally all of the NICs in the bond get marked as down and the events
process runs away with 100% CPU utilization on a single thread. Nothing can be done to bring the bond back up other than restarting the server; then everything goes back to normal.
The NIC in use is an Intel VT Quad Port Server Adapter
and all four ports are bonded. 802.3ad
has been correctly enabled on the switch (a Netgear Smart Switch) and functions properly.
Nothing is appearing in the logs after a reboot, although when examining the terminal after the crash/failure has occurred, error messages about not being able to reset the NIC have been printed. I cannot remember the exact message; I expected them to be in one of the log files so I just rebooted to bring the server back up.
Here is how the bond is configured:
############################
# modprobe.d configuration #
############################
alias bond0 bonding
options bond0 mode=4 miimon=100 arp_interval=100 arp_ip_target=10.1.1.1 max_bonds=2
# Another bonding rule for two other NICs...
############################
# relevent lines in #
# /etc/network/interfaces #
############################
auto bond0
iface bond0 inet static
address 10.1.1.100
netmask 255.255.254.0
broadcast 10.1.1.255
network 1.1.0.0
gateway 10.1.0.1
slaves eth2 eth3 eth4 eth5
bond-mode 802.3ad
bond-miimon 100
bond-downdelay 200
bond-updelay 200
dns-nameservers 10.1.0.1
In the above configuration, 10.1.1.1
, the arp_ip_target
, is the IP address of the Switch, which is always on.
Anyone seen this before or have any suggestions? It has happened about twice in the past two months; it is pretty hard to pinpoint exactly what is causing the problem, but it needs to be fixed. I've tried everything I can think of to reproduce the problem.
Update: One of the interfaces on NIC started only negotiating a 10mbit speed with the switch. I confirmed that cable wasn't the problem, nor is the switch by plugging a laptop with a 1gbit NIC directly into the interface on the server that is experiencing problems; it first negotiates a 1000gbit speed, a few seconds later dies, and re-negotiates a 100mbit speed. I also tried plugging another interface on the card in question into the same port on the switch the failing interface was plugged into and it works perfectly.
Assume for a second that only this one interface on the card is failing, not the entire card itself. Why would the entire bond fail if one interface dropped out? Is there something wrong with my configuration (above)? If not, I'm assuming that the entire card is slowly starting to fail and will need to be replaced.
Either way I plan on replacing the entire card, I just want to make sure that my configuration is correct for both future failures and educational purposes.
events
really run away with 100% of a core?