Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is it possible to span one huge Virtual Machine across several physical commodity servers?

Here is our use case:

  • We need to implement a 32-processor db server with 64 GB of RAM
  • We don't have a physical server of such capacity
  • We do have a lot of servers with smaller resources.

Is there a technology or (better) a product that lets us utilize these servers to create a VM with required capacity? Say, are we able to combine 8 physical 4-processor machines with 8 GB of RAM each into one 32-processor "logical unit" with 64 GB of RAM and set up an Oracle server that uses all this capacity?

Before posting this question, we read similar questions but didn't find an answer.

Maybe, someone could give us a hint now?

share|improve this question
1  
This is not an answer for your question, but It's feel strange that nobody advice about to look at the software limitations. If your company build apps for middle range business, it's seems to me obvious that the problem are software limitations probably the software architect and designers didn't think in a database with billions of records or with huge temp tables or procedures, think on that and create some self test and error reporting for the slow queries for my it's the way to solve the problem... think on the 3.3GB limit in the x86 –  user131482 Aug 8 '12 at 21:33

4 Answers 4

up vote 3 down vote accepted

There is a commercial product from ScaleMP called vSMP. It allows you to aggregate multiple x86 systems into a single virtual instance. I've never personally tried this before though, but I've been through a presentation from them. If I remember correctly, there are specific requirements for this to work, and you'll need to get some additional hardware (Infiniband for fast, low latency interconnects). It might cost a pretty penny too!

share|improve this answer
1  
ScaleMP doesn't emulate an x86 environment. You'll never get Windows or any other standard x86 OS to run in the virtual environment. You The only flavors supported are various versions of Linux built on SMP type architecture. And of that type of architecture... there are SEVERAL flavors. Even free ones. –  TheCompWiz Sep 20 '10 at 13:02
    
OP was not specific about the other requirements. I only answered what I could gather from his/her post. –  ryanlim Sep 20 '10 at 15:26
1  
This looks bloody cool. I suspect that a 32 core box (possible with 2x 16 Core AMD chips) might be cheaper than a cluster with Infiniband, but there we go. This solution earns more bragging rights. –  Tom O'Connor Aug 8 '12 at 21:53

There is no way to get the exact same functionality as a single 32-processor machine... with several separate servers. Your best bet is to look at clustering or grid computing. Done right, you can end up with comparable performance... and a higher level of high-availability. A lot of your question also depends on your "db" type. Microsoft SQL Server works significantly differently than MySQL or Oracle... and the scalability is also done completely differently.

Alternatively... you may want to consider letting someone do the database for you... like using the EC2 RDS...

Sadly, there is no way to combine several physical servers together & slap vmware on them and end up with a singular uber-powerful virtual server.

share|improve this answer
    
TheCompWiz, thanks for the answer. OK, if the answer depends on my db type, let it be Oracle or Microsoft SQL Server. With these corrections, is it still impossible? Yes, we know about EC2 but we need exactly Oracle or Microsoft SQL Server to test issues with a software product we deliver for a customer... –  user54614 Sep 17 '10 at 20:01
    
Also, why only vmware to take into account? We don't mind any other hypervisors... –  user54614 Sep 17 '10 at 20:03
    
The ability to traverse multiple servers is a HUGE logistics nightmare... not to mention the lack of available bandwidth between devices. Think about how fast a CPU is... then all of the things you would have to do that would slow down the process... i.e. CPU -> bus -> PCI-bridge -> network card -> ethernet cable -> network stack -> ... even before it's reached the other server? You wouldn't want to wait 1 second to be able to add 1+1. Clusters typically are able to do this because tasks are assigned in "Jobs" and a job is issued to a compute node which does all the tasks in that job... –  TheCompWiz Sep 17 '10 at 20:20
    
... and then sends the answers back to the management node. Windows doesn't. There is no way to setup a virtual X86 environment (or X86_64) that would even attempt doing this. –  TheCompWiz Sep 17 '10 at 20:23
2  
@user54614 - You're absolutely not going to be able to replicate their scenario by strapping machines together. I would suggest talking to both your client and Oracle support to pinpoint and identify the problems. –  Chris Thorpe Sep 18 '10 at 1:38

"TheCompWiz" answered your question usefully.

I'd still like to say that, yes, you could build a hypervisor which allowed a single VM to span several physical hosts and it could run that VM "correctly" where everything functioned.

But, even with really good, high-speed networks between the physical hosts, the performance of such a thing would be truly awful, running much more slowly than a smaller VM that fit within a single one of those hosts. You'd have to simulate the cache coherency properties of a single VM by intercepting every single memory read or write that the guest OS and application did, which would multiply the cost of memory access by thousands, if not millions.

So no commercial hypervisor vendor enables such a thing. It's been tried in the lab. Nobody has bothered to make a product out of it.

To underscore the point one more time, look toward clustering for a solution.

share|improve this answer
    
But what if a software product we deliver works fine for most customers but fails to work properly on the environment of a huge customer which runs our application on an 32-proc Oracle server with 64 GB of RAM. We want to reproduce this failure in our environment. –  user54614 Sep 17 '10 at 20:37
2  
I know nothing of your software, but what's happening at 32-processors and 64GB RAM that's not happening at 2-processors and 8 GB RAM? If there's truly something repeatably wrong at that level, then it's an Oracle/OS/driver/IO/hardware issue. –  gravyface Sep 17 '10 at 20:46
    
You will never get a hypervisor to traverse physical machines. They're still confined within the physical core of the machine. That being said... I bet you could build a mainframe type architecture similar to those of archaic behemoths from long ago... but you'd never get anything x86 running on it. –  TheCompWiz Sep 17 '10 at 21:00
    
Your huge customer should have a second QA instance of that monster database server. If they don't have that available, it truly is their problem. In 15 years of IT work, I have never seen anybody expect a software vendor to duplicate their infrastructure (unless it is part of a service contract specifying exactly that, and the customer pays for it). Especially when that infrastructure is esoteric (although a 32-core 64-GB server can be had for about $22K from Dell these days). –  rmalayter Sep 17 '10 at 22:03
    
What about ScaleMP (scalemp.com)? –  user54614 Sep 18 '10 at 13:54

Well well, what a lack of satisfactory answers here so let me add mine then...lol. It's the software that lacks threading over multiple physical units. As long it's within the local system the API (is it api?) can tell the software the amount of available processors the work with. Ans there's modern software that effectively spreads loads over the local cores and software that even can't properly balance the load over the local cores and then there's software who totally lack multithreading narrowing load to just one local core. So now, then there are the external cores that you want to share processing load to, start writing code that does that because as far as i know it's not implemented in commercial OS's. However there are applications who just do that and have cluster processing code inside for example MaxwelRender.

By the way, if a cpu is x amount of gigaherz it isn't really so. They already are hardwired as multiple central processing units of say 300 Mhz hence the multiplier number in CPU specs. Hmmm...that's probably not exactly true.

share|improve this answer
3  
Sorry, but I don't see how this answers the question? –  Dan Mar 27 '13 at 11:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.