There is a lot more than just ESXi in question here,
- Each VM will consume up to 4GBs + "overhead" which is documented here. This depends on the vCPUs, + memory allocated. At minimum each VM will use 4261.98 MBs (4096 + 165.98)
- ESXi's own memory overhead, this is hardware dependent. The easiest option is to look at the System memory usage in the vSphere client. From memory I recall it is around the 1.3GB mark, but as stated that is very dependent on hardware.
Memory Allocation & Overcommitment Explained
Note that the hypervisor won't allocate all of that memory upfront, it is dependent on the VM's usage. However, it is worthwhile understanding what will happen should the VMs try to allocate and use all of memory allocated to them.
The maximum your VM + host will try to use will be approximately, 55 GBs milage may vary
- 1.3 GBs used by ESXi
- 4261.98 MBs * 13 used by the VMs
There is another aspect to take into account and that's memory thresholds.
By default VMware will aim to have 6% free (high memory threshold). So the 55 GBs of used memory needs to be reduced down to ~45GBs
That means the host will have approximatley 10,500 MBs of memory it needs to reclaim back from somewhere should the VMs use the memory they've been allocated. There are three things ESX does to find that additional 10.5 GBs.
Memory Reclamation Methods
- Transparent Page Sharing
- Memory Ballooning
- Hypervisor Swapping
You should read and understand Understanding Memory Resource Management
in VMware® ESX™ Server.
Depending on a large number of factors, a combination of all three will / could happen on an over committed host. You need to test your envrionment and monitor these metrics to understand the impact of over committing.
Some rough rules that are worth knowing (all in the above paper and other sources).
- Transparent page sharing does not happen for VMs that use 2/4 MB pages. As you've allocated 4096 MBs to your Windows VMs, they will use the 2/4 MB pages by default (PAE dependent). Only under memory pressure will VMware break the large pages down to 4 KB pages that can be shared. TPS relies on using idle CPU cycles and scanning memory pages at a certain rate. It returns memory relatively slowly (think an hour rather than minutes). So a boot storm will means TPS will not help you. From the three, this has the lowest performance impact. More from the document,
In hardware-assisted memory virtualization (for example, Intel EPT
Hardware Assist and AMD RVI Hardware Assist [6]) systems, ESX will
automatically back guest physical pages with large host physical pages
(2MB contiguous memory region instead of 4KB for regular pages) for
better performance due to less TLB misses. In such systems, ESX will
not share those large pages because: 1) the probability of finding two
large pages having identical contents is low, and 2) the overhead of
doing a bit-by-bit comparison for a 2MB page is much larger than for a
4KB page. However, ESX still generates hashes for the 4KB pages within
each large page. Since ESX will not swap out large pages, during host
swapping, the large page will be broken into small pages so that these
pre-generated hashes can be used to share the small pages before they
are swapped out. In short, we may not observe any page sharing for
hardware-assisted memory virtualization systems until host memory is
overcommitted.
Ballooning kicks in next (thresholds are configurable, by default this is when the host has les than 6% memory free (between high and software)). Make sure you install the driver, and watch out for Java and managed applications in general. The OS has no insight into what the garbage collector will do next and it will end up hitting pages that have been swapped to disk. It is not uncommon practice for servers that run java applications exclusively to disable swap entirely to guarantee that doesn't happen. Have a look at Page 17 of vSphere Memory Management, SPECjbb
Hypervisor swapping, from the three methods is the only one that guarantees "memory" being available to the hypervisor in a set time. This will be used if 1 & 2 do not give it enough memory to remain under the hard threshold (default of 2% free memory). When you read through the performance metrics (do your own), you'll realise this is the worst performing of the three. Aim to avoid it at all cost as the performance impact will be very noticable on nearly all applications double digit percentage
There is one more state to be aware of low (by default 1%). From the manual this can drastically cut your performance,
In a rare case where host free memory drops below the low threshold,
the hypervisor continues to reclaim memory through swapping and memory
compression, and additionally blocks the execution of all virtual
machines that consume more memory than their target memory
allocations.
Summary
The key point to stress is it is impossible to predict from the whitepapers how your environment will behave.
- How much can TPS give you? (Depends on how similar your VMs are with their OS, Service Pack, and running applications)
- How quickly do your VMs allocate your memory? The quicker they do, the more likely you are to jump to the next threshold before the less impactful memory reclamation scheme succeeds in keeping you in your current threshold.
- Depending on application, each memory reclamation scheme will have widely varying impact.
Test your average scenarios, you're 95% percentile scenario, and finally your maximum to understand how your environment will run.
Edit 1
Worth adding that with vSphere 4 (or 4.1 can't recall), it is now possible to place the hypervisor swap on local disk but still vmotion the VM. If you're using shared storage I strongly recommend you move the hypervisor swap file to be on local disk by default. This ensures that when one host is under severe memory pressure, it doesn't end up impacting all the other vSphere hosts/VMs on the same shared storage.
Edit 2
Based on comments, made the fact that ESX doesn't allocate the memory upfront in bold...
Edit 3
Explained a little more about memory thresholds.