Every physical machine is connected to a cisco switch and is on the same network 192.168.1.0/24
Every server I have is running Centos 6.5. A couple run KVM with a bunch of virtual machines. Most of the machines have bonded nic's.
virtual server (vs0) has a bunch of vm's (vm00, vm01)
virtual server (vs1) has a bunch of vm's (vm10, vm11)
then I have a some other random physical machines (web0, web1)
Normally, everything works fine, but occasionally, certain machines will lose connectivity.
web0 cannot ping vm11
web0 can ping web1, vs0, vm00, vm01, vs1, vm10
vm11 cannot ping web0
vm11 can ping web1, vs0, vm00, vm01, vs1, vm10
All other machines can ping every other machine
And then 30 minutes later, everything is fine again.
Occasionally, web0 can't see web1, so I don't think its a KVM thing. And when web0 can't see vm11, web0 CAN see vs1, so I don't think its a nic issue.
I tried tcpdump on the machines and no packets are hitting the machines except ARP packets. I don't know what else to look for. I'm out of ideas for debugging this. Any ideas?