Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a large Xen deployment running on both RHEL and CentOS and have recently started looking at KVM since this is where it looks like the future of VM's are on Linux.

We can load the server and get everything running without an issue. However when we load up a new one with JBoss (4.2 Community edition, Sun JDK 6) and load a large EAR the server goes a little crazy. The %sy will jump to 80-99% and just hang for large periods of time we see a similar jump in %us on the host machine. We though at first this might be I/O as it seems to happen at start of JBoss but then would "cool down" after everything got loaded. We did some tests by extracting some large tar.gz files and using jar -xvf on the ear but could not re-create.

Then we starting thinking this might be some type of memory access issues. We loaded a c-program that would generate a lot of memory activity and sure enough we saw the spikes again. Not as high mind you but we did see it jump. We then wrote a small java program to do the same thing and sure enough we saw it jump again.

Any thoughts on what might be causing this? Is this just the way KVM works?

As a side note we do NOT see this behavior on any other setup. Xen, VMWare and/or native iron. The system does seem a bit slower than our Xen / VMware ones.

share|improve this question
    
sounds like an 'interesting' problem. Can you generate any more logging , traces, debug from both the guest and the host. How much memory do you have? how much do you allocate to kvm? are you paging? Is kvm ballooning memory on the fly? virtualization.info/2008/08/… Depending on how you kvm memory is configured. Worth a look. –  The Unix Janitor Mar 18 '10 at 14:35
1  
I don't want to discourage questions like this from being asked on this site but given that both KVM and JBOss are owned by RedHat I would contact their support if you have a support contract. Or also try sending an email to both the JBoss and KVM mailing lists. –  3dinfluence Mar 18 '10 at 14:48
    
I have contacted RHEL support and they are not much help to this point. Same issue presents on CentOS 5.4 but have not tried on a different distro like Ubuntu. We are also able to re-create the issue without JBoss with a C program and and a Java program that chews up memory. The ballooning memory was a thought too. I'll have to read through the link. –  telstone Mar 19 '10 at 12:40

4 Answers 4

up vote 1 down vote accepted

A question and a suggestion:

  1. What file system re you using? On Fedora 12 & 13 systems I have seen excellent performance using ext4, but abysmal performance using btrfs.

  2. Extending on @Ophidian comment - try it with Fedora 13 to see how it runs using recent KVM and libvirt libraries.

share|improve this answer
    
Upgrading to the most recent 5.5 versions of CentOS has seen most of the issues concerns go away. Testing with RHEL 6 shows excellent performance. We have not tried ext4 but we have use XFS and saw a nice jump in performance. –  telstone Feb 18 '11 at 21:38

I'd like to see the testing code you used to recreate this, if possible. I'm testing a lot of KVM guests all the time, and an extra benchmark is always good, especially if it's known to be causing problems

Knowing the VM config will help as well - how much RAM, CPUs, disk and network types etc

share|improve this answer
    
I'll get you the files we used to re-create the issue. I think we have a C and a Java version. We were going to cross compare with Xen just to make sure it would behave differently. –  telstone Mar 19 '10 at 12:38
    
That's perfect, if you want to refrain from posting the code here, just drop it to my nick @gmail Thanks :) –  dyasny Mar 19 '10 at 14:38

I wonder if you're somehow hitting swap on the host machine.

Have you / could you try this on Fedora 12 to see if the more recent versions of everything (kvm, libvirt, kernel, ksm) exhibit this same behaviour? KSM was added in Fedora 12 and purportedly provides much more efficient/better memory management.

In addition to 3dinfluence's suggestion to contact Red Hat support and the KVM mailing list, I would also suggest the fedora-virt list as a large number of developers for libvirt, kvm and the rest of the Fedora virtualization ecosystem are regular readers of the fedora-virt. They are generally very responsive and helpful.

share|improve this answer
    
thanks I will give that a shot. I have already contacted RHEL support but since JBoss start up normally (without a deployment) they are not being much help. We are working on pulling a stand alone example together and perhaps then we will get some help. The problem exists on CentOS as well. As would be expected. –  telstone Mar 19 '10 at 12:37
    
No swap on the host. Host has 72Gig of RAM with Dual Quad Core Intel Processors. VM is 10GB with 2 processors –  telstone Mar 19 '10 at 12:43

Are you running the ksmd (the shared memory de-duper)? This can have a bit of CPU overhead on the hypervisor system, and could conceivably be involved in memory-related stalls.

It may also be worth seeing if pinning the VM to a core helps - you may find that the guest being switched from core to core (or across processors) is causing cache flushes under memory-intensive workloads that screw things up.

Bearing in mind that every VM is a process as far as the hypervisor is concerned it could be illuminating to attach strace to the guest process and see if there are any syscalls that are obviously problematic.

Finally, I've been finding Dag Wieer's dstat a nice addition to my toolkit for tracking down problems.

(Since we're about to look at piloting JBoss-under-KVM this is relevant to my interests...)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.