My Xen HVM machine has 4 cores on 2.4 Ghz however top gives me a load of 2-3. There are running 20 VM's on it (most of them are idle). The HDD is also almost idle (200 KB/s write and 6 KB/s read).

xentop gives me Domain-0 -----r 223766 121.0 4376576 26.1 no limit n/a 4 25 6105332 2824789 0 0 0 0 0 (so 121.0 as CPU% usage).

What's the reason of this? None of the VPSs have high CPU usage. It can't be the IO.

Anyone with some more experience know how to pinpoint this problem?

Edit: You can find the output of top here: http://dl.dropbox.com/u/6166898/top.txt And here's the xentop full output: http://dl.dropbox.com/u/6166898/xentop.txt

There are some Windows VM's on it, I'm using Xen in conjunction with SolusVM.

-
xentop isn't going to shed any light on this. Include the output of top so that we can see what Dom0 is up to. –  3dinfluence Dec 2 '11 at 2:56
@3dinfluence I've added the top output. As you can see nothing wrong however the load is 2.49. –  Devator Dec 2 '11 at 10:25
Very strange. I have a quad core box running 7 light duty vm's on Xen. The Dom0 is showing almost no activity or load in top or xentop. Does vmstat show anything out of the ordinary like a high number of context switches? –  3dinfluence Dec 2 '11 at 15:09
@3dinfluence not sure what those numbers mean, but here is the output: dl.dropbox.com/u/6166898/vmstat.txt Thanks for sharing some of your time ;-) –  Devator Dec 2 '11 at 15:23
That all looks fine to me as well. Hopefully someone else will have some more ideas for you. –  3dinfluence Dec 2 '11 at 18:10

The high CPU load has been fixed by updating to Xen kernel 3.4.3 (from kernel 3.1.2).

Thanks to those who tried to help me!

-
Great you got it fixed :) Thanks for the bounty award. –  George Tasioulis Dec 5 '11 at 20:55

Open-source Xen doesn't provide drivers for Windows PV (at least not officially... I guess you could try those Open Source Windows PV drivers for Xen or take a look at this blog post on how to install Windows 2k8 R2 + the GPVPV drivers - the information there could help you). Btw what versions of Windows are you running on your Windows VMs?

One thing you can try though is go into every Windows VM's Device Manager and change the HAL to ACPI Multi and force reboot.

Finally please post the output of ps fawwux to see a list of processes and work out what's using the CPU utilization.

PS: If you want to provide Windows VPS with Xen, use XenServer Express (free) or Enterprise () and use the bundled PV drivers.

-
I'm using Xen HVM, not PV. However I will try to change HAL to ACPI and see if that works. –  Devator Dec 4 '11 at 13:43
Oh OK, then please try the thing with ACPI Multi and let us know :) –  George Tasioulis Dec 4 '11 at 15:51
The ACPI is no longer in Windows 7 it seems. Also, would the only thing to fix this be on the "client" side rather then on the server side to fix this? –  Devator Dec 4 '11 at 15:58
I've saved all the Windows machine on to disk (so they are no longer active and the load goes back to around 1.5 and now it's rising again to 3.5. It seems the load is lower now, however still not stable at all. I'm not sure if it's the Windows now (I don't think so, since they are not even online anymore). –  Devator Dec 4 '11 at 16:19
Could you post the output of ps fawwux ? –  George Tasioulis Dec 4 '11 at 16:45

Perhaps this is massively wrong, but given that you've got 4 cores, and the load average is a measure of number of processes waiting for CPU time, perhaps there's actually no problem.

Have a read of Understanding Load Averages, I suspect that there's not actually any major problem.

You haven't mentioned whether the server or it's VMs are misbehaving, or if it's just that the load average is higher than you'd expect. If it's just the latter, you might want to review and readjust your expectations.

From the vmstat output, the host looks happy, the IOWait is low, the load average is less than the number of cores. You might want to fiddle with CPU affinities, but I'd say that it's probably actually performing OK, from the information you've provided.

-
Unfortunately, the load even reaches 7 or 8 which will give connections errors from the control panel (SolusVM). So it's actually misbehaving :(. –  Devator Dec 4 '11 at 16:48
Do you pay SolusVM for support? Could it be that their product is just pap? –  Tom O'Connor Dec 4 '11 at 16:49
There are are so many people using SolusVM, I doubt is their product but I will keep it in mind obviously. I don't get your latter question though. What do you mean with it? –  Devator Dec 4 '11 at 16:53
Could you possibly run ps ef on the vm host, and pastebin that output? –  Tom O'Connor Dec 4 '11 at 16:55
Here you go: pastebin.com/7yZMpVr9 –  Devator Dec 4 '11 at 16:57