Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Every physical machine is connected to a cisco switch and is on the same network 192.168.1.0/24

Every server I have is running Centos 6.5. A couple run KVM with a bunch of virtual machines. Most of the machines have bonded nic's.

Machines:

virtual server (vs0) has a bunch of vm's (vm00, vm01)

virtual server (vs1) has a bunch of vm's (vm10, vm11)

then I have a some other random physical machines (web0, web1)

Normally, everything works fine, but occasionally, certain machines will lose connectivity.

web0 cannot ping vm11

web0 can ping web1, vs0, vm00, vm01, vs1, vm10

vm11 cannot ping web0

vm11 can ping web1, vs0, vm00, vm01, vs1, vm10

All other machines can ping every other machine

And then 30 minutes later, everything is fine again.

Occasionally, web0 can't see web1, so I don't think its a KVM thing. And when web0 can't see vm11, web0 CAN see vs1, so I don't think its a nic issue.

I tried tcpdump on the machines and no packets are hitting the machines except ARP packets. I don't know what else to look for. I'm out of ideas for debugging this. Any ideas?

share|improve this question
    
Have you checked for error stats on the switch itself? –  mfinni May 2 at 19:08
    
I don't know exactly what I'm looking for, but the switch has 0 drop events, CRC & ALign Errors, Undersize/Oversize Packets, Fragments, Jaggers, Collisions (from Status and Statistics->RMON->Statistics). I'll continue to look for additional information in the switch though. –  dan May 2 at 19:16
    
Under View Log->RAM Memory, there is nothing from May except my login. I see some link up and link down entries from a while ago, but I assume those are from me plugging stuff in. There are no entries during the time period today when the machines lost connectivity. –  dan May 2 at 19:21
    
From one server that web0 can ping, and one it can't, use arp to check what it thinks the MAC address for web0 is. I suspect its an arp issue caused by the bonding. –  Grant May 2 at 21:52

1 Answer 1

I wonder if the problem relates to the way the bonding is being done, and the switch getting confused with MAC addresses on ports etc.

I'm guessing you are using "Mode 0" for bonding. Try changing things to Mode 1 (Active-backup), or (temporarily) dropping one of the underlying bonding interfaces on each device and see if that makes things more stable.

share|improve this answer
    
I'm actually using mode 6, BONDING_OPTS="mode=6 miimon=100", but I will try dropping a connection when I get back to work on monday and see what happens. –  dan May 4 at 14:44
    
annoyingly, I couldn't get it to break again so I could look at arp values. i went ahead and just updated the bonding mode to 5. bugzilla.redhat.com/show_bug.cgi?id=584872#c16 –  dan May 6 at 18:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.