Not advisable to run ESX 4.0 host with 128Gb RAM - is there any truth to that?

Question

I was advise by my webhosting company providing managed colocation dedicated server service that adding 128Gb RAM to a Sunfire X2200M2 Server (Dual Quad Core 1.9Ghz) to run ESX 4.0 Host would not be advisable.

The Sunfire server support a maximum 128GB RAM, but they say its no point adding that much RAM as the Dual Quad Core CPU 1.9Ghz won’t be able to cope with so much RAM in a virtualise environment and they recommend 24GB as the maximum. Anything higher, will experience performance degration. Does it make any sense to you?

The maximum supported RAM for ESX4.0 is 1TB as stated in the configuration maximum doc: http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

The following shows that 128GB host has been deployed on HP hardware, so why not Sunfire?? http://www.eliaskhnaser.com/Article.aspx?blog=2&id=33

Any ESX guru out there like to share their thoughts? Thank you.

Answer 1

According to Sun's specs the max RAM it'll take is 64Gb. That said, there's no reason not to take ESXi or that server to its max ram. Unless your colo techs can come up with a very specific reasons. We're running ESXi servers quite happily with >100Gb of RAM on Dell and HP hardware, so unless there's some technical limit on that particular server model, I'd say you're good to go. The server is on VMWare's ESXi Hardware Compatibility List with a max RAM listing of 64Gb.

In my experience, with 2008 becoming more common as a VM Guest OS, you almost always run out of RAM on a virtual host before you run out of CPU. It depends on workload, of course, but VMs LOVE RAM, and this trend seems to be getting more and more pronounced.

I would warn that 8Gb RAM sticks are uneconomically expensive (3x or more the cost of 4Gb sticks). 4Gb sticks seem to be the sweet spot.

Another suggestion: If you're considering taking a single host to such a high level of RAM and this is your only ESXi host, you are putting a lot of eggs in 1 basket. A good option that gets you both scalability and resilience is to buy a second identical server and run them as a pair with less ram per server... Then if your one server falls over, you won't lose your environment.

EDIT: Just to illustrate the CPU-vs-memory usage in a production environment, here's a snippet from an email I sent out a few days ago, showing CPU and RAM utilization on virtual hosts at one of our datacenters. The servers in question are HP DL360 G6 dual-quads with 70Gb RAM per host, running ~160 VMs across the 5. I'd love to see if others are seeing similar trends, or if this is just a characteristic of our workloads.