Virtual machines are not like typical servers, in that you run into problems in different areas. Most of the time, CPU isn't the bottlenecking resource, but RAM is. The things to really know before you go in:
- Disk throughput How fast do you pound your storage? MB/read, MB/write both average and peak (as mentioned elsewhere in this thread RRDTool is good for this). Do you know when your peaks are, and whether or not they'll coincide with I/O peaks on other VMs stored on the same ESX cluster. In our environment backups are the peak I/O time, but we get bursts during the day. The answer to this will tell you whether or not you can get away with file-backed disks, or if you have to direct present LUNs to VMs.
- Network throughput Know how fast you need to be. As above, backups are the area when we start attempting to saturate our NICs. Know how much data you're pounding out. I'm pretty sure there are NICs out there that can do VLAN tagging, which can ease load-balancing problems if your network infrastructure supports it.
- RAM creep Known your programs. We have one that will consume every bit of memory given to it, which causes the VMWare console to whine and complain about usage and recommend giving it more. If you're not as tragically underfunded as we are, hopefully your ESX servers will be provisioned with a lot of RAM. In our environment, we consider a VM to be 'piggy' if it needs over 1GB of RAM. Yours may be different.
Determining whether or not you can use file-backed disks or if you require direct-presented LUNs can take a bit of knowing. Direct-presented LUNs are where your storage array presents LUNs directly to VM's, which is made easier by using NPIV. You can do it without NPIV but it may be too perilous for your blood, all brand new Fibre Channel hardware should support it and ESX 3.5 certainly does. Direct presented removes a layer of abstraction between the storage array and the virtual machine pounding I/O, and in that sense it can provide better performance. However, direct-presentation is trickier to set up and has a higher start-up time in the "wrap your head around it" stage.
File backed disks are just plain easier. Plus, they can be moved between storage arrays pretty simply (for certain values of simple, where copying multi GB files is concerned), something that direct-presentation requires (usually very expensive) array-level replication software to accomplish. Low I/O load things work just peachy on file-backed, and even some higher I/O things as well. We're running a full Exchange 2007 install for over 3000 users on file-backed disks. The backups could be faster, but during the day the users don't notice any slow-downs.