Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Scientific Linux 6.3 KVM host running 2 nearly identical Ubuntu 11.04 guests. I built 1 base image and then cloned. Each vm has mysql and different RoR app installed on apache, using nginx as reverse proxy (as requested by the devs).

This setup was working 100% for months until about a week ago, then I started noticing that the networking on one of the vm was dropping temporarily. Specifically, by port-scanning (nmap) the guest I can see that the https port (443, the only non-filtered port except for ssh) first goes closed, so the firewall is still letting packets through but there is no longer a service listening and then between 5 -20 secs later the port is filtered but the host still responds to ping. Finally a few seconds later it stops responding to pings and seems to drop network completely. 10 - 30 seconds later everything is back up as if nothing ever happened. Sometimes this whole process happens every 2 minutes of so, other times seems to be up to 20-30 mins between interruptions.

I can ssh into the box, but when network drops I lose connection. There is noting in dmesg or syslogs on the guest or the host to suggest anything is askew.

Almost the strangest thing is that the ssh connection seems to miss character presses. Sometimes I'll hit enter and nothing happens the hit it again an 2 go through at the same time. Also if I cat dmesg or syslog sometimes it just stops randomly midway through the file, in the middle of a line. Sometimes ssh connection drops with a "Write failed: Broken pipe" when I hit a key (this seems almost independent of the network interruptions).

Also, this is happening on only 1 vm, the other is 100% normal. The vms are both using virtio driver and guests have different mac addresses. All software has been updated on both vms and host.

I'm fairly sure this is a problem on the host or guest and not the network or firewalls as I'm testing this both from my workstation and from ssh on the host itself with identical results, so I would guess the problem is in the guest. The guest is never using more than 20-30% ram and cpu and disk space on all mounts is below 50%.

I am completely confused and have no idea how to debug further.

share|improve this question
    
Are both VMs using the same network bridge and physical ethernet port on the host? If they are it could be a layer 2 / port blocking issue. If not, it may be a bad port / cable on the misbehaving VM. –  AngerClown May 30 '13 at 18:27
    
Yes they are using the same physical ethernet port. Could you explain what could cause a layer2 / port blocking issue? –  Jacques May 31 '13 at 8:05
    
I was wondering if it was something with the switch seeing multiple MACs from the same port. Usually though port security it either working or not though, but still worth checking with the switch admins. –  AngerClown May 31 '13 at 12:14
    
Does the VM actually freeze, or just the network hangs? Try to log into the VM through a VNC/Spice/serial console while the issue occurs –  dyasny May 31 '13 at 15:11
    
@AngerClown, no new switches have neen installed or anything and was working for months in current setup. We have 10-15 other similar vm hosts running similar setups working fine. –  Jacques Jun 1 '13 at 14:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.