I am experiencing a extreme strange problem...
We got a Dell R410 Host with Xen 4.1 installed(Ubuntu 12.10). In this host, we created 6 VMs, one of them is a HVM Windows and others are all PV Debian-like systems.
There are 2 NICs on this host, all VM are bridged to xenbr1 with em1.
When our own staff try to access the websites which hosted on a VM, everything works well. No error, no latency. However, when I modify our firewall to allow all customers to access this VM, the host and VM quickly become inaccessible. When I try to ping all VM and host from firewall, the result is extremely unstable -- most packets are lost, few are successfully returned.
If I deny all incoming connections from firewall, both VMs and host become accessible again, no packet loss or latency.
The NICs of our host are:
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20) 01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
Our overall bandwidth is 50MB from ISP. And in firewall, I can see the amount of dynamic NAT rules is <400.
I also found, when all VM and host lost connection, the VM's load is incredibly high(>40). But host's load is normal(<3)
Why this happened? I completely have no idea.