I have a Scientific Linux 6.3 KVM host running 2 nearly identical Ubuntu 11.04 guests. I built 1 base image and then cloned. Each vm has mysql and different RoR app installed on apache, using nginx as reverse proxy (as requested by the devs).
This setup was working 100% for months until about a week ago, then I started noticing that the networking on one of the vm was dropping temporarily. Specifically, by port-scanning (nmap) the guest I can see that the https port (443, the only non-filtered port except for ssh) first goes closed, so the firewall is still letting packets through but there is no longer a service listening and then between 5 -20 secs later the port is filtered but the host still responds to ping. Finally a few seconds later it stops responding to pings and seems to drop network completely. 10 - 30 seconds later everything is back up as if nothing ever happened. Sometimes this whole process happens every 2 minutes of so, other times seems to be up to 20-30 mins between interruptions.
I can ssh into the box, but when network drops I lose connection. There is noting in dmesg or syslogs on the guest or the host to suggest anything is askew.
Almost the strangest thing is that the ssh connection seems to miss character presses. Sometimes I'll hit enter and nothing happens the hit it again an 2 go through at the same time. Also if I cat dmesg or syslog sometimes it just stops randomly midway through the file, in the middle of a line. Sometimes ssh connection drops with a "Write failed: Broken pipe" when I hit a key (this seems almost independent of the network interruptions).
Also, this is happening on only 1 vm, the other is 100% normal. The vms are both using virtio driver and guests have different mac addresses. All software has been updated on both vms and host.
I'm fairly sure this is a problem on the host or guest and not the network or firewalls as I'm testing this both from my workstation and from ssh on the host itself with identical results, so I would guess the problem is in the guest. The guest is never using more than 20-30% ram and cpu and disk space on all mounts is below 50%.
I am completely confused and have no idea how to debug further.