3

At Travis CI (http://travis-ci.org) we use VirtualBox VMs (through Vagrant) for running tests for the Ruby community.

On our worker servers we have up to (up to) N parallel processes running N test suites in N VMs in parallel, i.e. one worker process runs one test suite in one VM at a time, but N of them are running concurrently.

Now, as soon as many workers are actually performing builds in parallel the performance of each build will degrade significantly compared to when the very same build would be run in a single worker (and nothing else running in parallel).

Here is an example:

This "build matrix" consists of 20 individual builds:

http://staging.travis-ci.org/#!/svenfuchs/rails/builds/1906

At the time when this was run there were 10 workers running, so this build started out with 10 individual builds being executed in 10 workers (and VMs) in parallel. This build is one of them and it has taken ~ 2 hours to complete:

[see the last link in the list on the page above, i only can post 2 urls]

The very same build would take only ~ 20 minutes when there are no other builds being executed in parallel. Here's an example of that:

http://staging.travis-ci.org/#!/svenfuchs/rails/builds/1927

This performance degradation obviously is something we need to sort out but we're not sure where to look.

The test suite basically executes Ruby processes which might shell out and spawn several other Ruby processes each executing unit tests on the codebase. Some of them hit databases such as MySQL, Sqlite3 and Postgres but we also notice the same sort of degradation with tests that do not hit any database at all.

The worker server that hosts these processes and VMs looks like this:

  • Linux 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux
  • 12x (Hexacore) Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
  • 12 GB Memory

Each VM:

  • Linux lucid32 2.6.32-28-generic #55-Ubuntu SMP Mon Jan 10 21:21:01 UTC 2011 i686 GNU/Linux
  • 1x Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
  • 1 GB Memory

Any hints on how to sort this out or maybe just better identify the root problem would be highly appreciated.

Thanks!

4 Answers 4

8

You don't mention what the underlying disk is like on these servers, but these type of performance issues are nearly always IO-related. What are the IO stats looking like when you're running multiple builds versus just one?

Additionally, you'd get much better performance out of your hardware by using something like Xen or VMware ESXi as opposed to VirtualBox.

3
  • I don't think ESX or Xen are an option right now, because Travis depends on Vagrant for the deployment process.
    – Koraktor
    Jul 12, 2011 at 12:52
  • @Koraktor - fair enough, I guessed as much. I just wanted to do my due diligence and mention the other options, since the additional performance overhead of VirtualBox is likely affecting his system.
    – EEAA
    Jul 12, 2011 at 13:51
  • Sure... just wanted to note that to prevent further comments on VirtualBox being not the right choice. ;)
    – Koraktor
    Jul 12, 2011 at 14:14
2

did you ever try to modify the following vm.attributes?

attribute :cpus

attribute :synthcpu

I ran once in problems during heavy parallelization on a VirtualBox vm, which actually used more threads than could be served by the host and configured vm. Reducing the count of virtual cpus helped me that time.

This might be a hint where to look..

References to the attributes can be found there:

config.vm.customize: http://vagrantup.com/docs/vagrantfile.html VM Attributes: http://mitchellh.github.com/virtualbox/VirtualBox/VM.html

Good luck.. :)

1

I am a little confused on the server. The server is a 12 cpu x i7-950 quad core cpus? What box can do that? If it exists, I want one! You also mention hexacore, but the 950 part is a quad core. How many real cores do you have available? As I have never heard of a 12x board for i7, I am assuming you mean you have 2 i7-970(which are hecacore) for a total of 12 cores and 12gb ram.

Your problem looks pretty clear to me. You are running the OS + 10 VMs and each VM is allocated 4 cores. That’s 40 virtual cores plus another 1, at least, for the OS on a box that has 12 physical cores (24 with hyperthreading). I would expect 2 VMs to work well on this configuration, and then to see serious degradation beyond that.

If you do actually have what you describe (12 x i7-950) that is 48 real cores, in which case, the issues is ram ad i/o.

I don't have any specific virtualbox experience, but I do know vmWare and hardware.

Good luck

0

I too would suspect disk io. However you didn't mention what checks you already performed for this. top? vmstat? iostat?

It might be interesting to plot the cpu load per number of virtualboxes and see if there is a sudden performance drop. Maybe with two or three you do get a linear performance increase, but as soon as you hit ten it drops. This could very well be caused by memory io as well, which has the same problems as disk io, albeit on a faster speed: The software might not all fit in the cache anymore, so it has to get its stuff from memory, which is shared by 12 other cpus.

just my 2 cents.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .