I am encountering strange issue. I've several xen virtual machines deployed at HP Proliant Servers (120,360,380 gen5,6,7 - different). Some of them are PVM XEN running linux guest, some of them running XEN HVM windows server. Network is bridged. In general linux pvm work ok for years, no matter if host server is overloaded, lacks memory - everything is then clear and no problem to determine reason of unstable operation.
However problems sometimes occurs with windows hvm - from time to time (it depends - it may be months, days or even hours) there is connection loss to VMs. Configuration is as follows:
router -> switch -> physical interface -> bridge -> virtual interface -> VM
So it is typical bridged configuration. When problem occurs lets trace 'PING": (it was verified by tcpdumps at all above nodes):
router (REQuest) -> switch (REQ) -> physical interface (REQ) -> bridge (REQ) -> virtual interface (REQ) -> VM (REQ) and it REPlies -> virtual interface (REP) -> bridge (REP) -> physical interface (REP) -> switch (no reply) -> router (no reply)
So it looks like there is a problem at physical interface - as you can see something wrong with packets that should be transmitted to lower layers between physical interface and medium (patchcord) - please NOTICE that it occurs at the SAME TIME at different host servers! They are different series HP servers. What can I do to investigate it deeper? There is absolutely no interface errors at router,switch, bridge, interfaces. Something must go wrong at lower layers of physical interface. OSes are Centos 5.x - different kernels at the same time on different machines. It looks like problem is only related to HP Server with HVM Windows. PVM work without problems. Any clues? What can I also check to find a faulty place? To sum up:
- different switches, cables, network interface
- different machines (however, all HP Proliant series)
- same vlan
- different centos host os - different kernels
- same guest os - windows server 2008r2 - different updates installed.
- it looks like traffic independent - I mean MB/s and frames per second independent.
- no interace errors at all nodes
- no host server LOAD, memory lack
- SAME TIME traffic loss at different host nodes
- different network card firmware
Last days problem occurs several times per day. It would be great if you could give me some clues what can I also check to determine problem origin.