2

I have a VMWare ESxi box, 22GB ram, Dual Quad Core Xeon, 2 Sas drives + Write caching raid controller etc.

Anyways, have about 30 small XP VM's running on it and starting to get some very slow boot times and other performance issues. I THINK its I/O but looking at the graphs not too sure what to look for. Any ideas on what to look for would be appreciated. Here is the data I've got so far:

(I feel like my IO is high but not sure what to bench it against)

Disk IO Memory Usage enter image description here

Improve this question
4
  • Can you expand a little more on what the setup is - i.e. how may vCPUs etc. It looks like all your machines are on the same vmfs volume. Since you have two drives I'm assuming a RAID 1 setup - correct me if i'm wrong.
    – Zypher
    Feb 5, 2011 at 3:51
  • Yup, its raid 1. I am not sure what you mean by vCPUs but in configuration have 2 sockets, 4 cores per socket and 16 logical processors.
    – nextgenneo
    Feb 5, 2011 at 3:52
  • .. by vCPU's i mean how many CPUs do you have assigned to each VM?
    – Zypher
    Feb 5, 2011 at 4:05
  • I have 1 vCPU assigned to each VM and each VM has about 500MB Ram reserved.
    – nextgenneo
    Feb 5, 2011 at 4:13

2 Answers 2

Reset to default
7

30 VMs served from just 2 spindles (disks) will probably suffer an IO bottleneck, even if those VMs aren't particularly IO intensive (either random or sequential). You're looking at 30 separate concurrent read requests occurring across widely separated areas of the disks. Lots and lots of time wasted seeking between places.

I'd recommend setting up a second drive array if the option is easily available to you (spare drive slots or a spare external housing), and migrating your VMs across to it. 4-6 disks min. Another improvement would be a larger read/write cache, if you're only running on a 128 or 256 chip.

Another place to check is the vCPU allocations as Zypher mentioned - assigning too many vCPUs to each VM is (counter-intuitively) likely to slow all the VMs down (each VM has to wait for a free core for every single one of its vCPUs before it can get CPU time, so a 4vCPU VM may get less cycles than a 2vCPU VM)

Edit: thinking about it a bit more, there are also some locking problems you might come across by having so many VMs on a single LUN. You can encounter per-datastore locks during various VM operations, possibly power-on/suspends etc. That'll start to stack up quite quickly so slow boot-ups etc may be caused by this. You can get around this by setting up separate datastores within the same amount of drive space (resize the current partition to half, then create a new partition in the blank space. Spread VMs evenly between the datastores). About 15 Vms per datastore is a good maximum.

Improve this answer
3
  • Yeah, i think its not a vCPU issue as we have 1 per VM. Looking at read latency it definately looks like that would be the issue. So you'd rec one drive per 4-6 VMs?
    – nextgenneo
    Feb 5, 2011 at 4:31
  • Also what are your feelings on SATA vs SAS vs SSD for this sort of setup?
    – nextgenneo
    Feb 5, 2011 at 4:40
  • SSDs can work well if you can stretch to the price. SAS is a bit faster than SATA, and usually spins faster which helps enourmously. SSDs are still way out ahead of the pack if you're constrained to 2 drives.
    – Chris Thorpe
    Feb 6, 2011 at 10:37
0

You want to look at esxtop and read the manual In particular section 4.2.2 Latency Statistics and 4.2.3 Queue Statistics.

You could benchmark your esx store (run iotop in a few VMs at once) to see what kind of read/write disk rates those graphs give you. You really can only compare you storage against itself. Watch esxtop at the same time, to see how DAVG and QUED change.

I'm not surprised 30 VMs are becoming slow on two disks.

As for SATA vs SAS vs SSD, that's basically their order of speed and cost. So if you have the money and need the speed, move to SAS or even SSD. But you can also look at raid 10 of SATA disks first.

Improve this answer
2
  • in terms of drives if we are running 6 machines on each drive, don't need much hdd space so a 64gb ssd would be affordable and fast so this would make a lot of sense. any experience with how well ssd's perform on esxi?
    – nextgenneo
    Feb 5, 2011 at 21:19
  • They perform well. There is no TRIM support, so you'd want to look at a drive which handles that internally. One example: anandtech.com/show/4010/kingston-ssdnow-v-plus-100-review/9
    – Steven
    Feb 7, 2011 at 17:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .