Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I put together an ESX box for better management, but its performance is a WTF item; I'm a noob at dealing with ESX, so I'm looking for a laundry-list of reading material to help me straighten this out so I can go back to .NET programming.

Current storage system:

We're running Raid5+Hotspare (8x500 GB spindles) on a PERC6i on a Dell 2910. Due to ESX limitatios, the PERC is showing the storage as 1x2TB + 1x800GB "partitions." I'm not sure of the setup's configuration (stride / stripe / ???) at all.

Our Applications

We have a SBS server as well as a minor (2x50 GB, but growing at 10GB/month) database server...

Our application that lives on the database VM is CPU and I/O insense; it's a database churning excercise mixed in with a lot of computation on the data (fixing that performance is what I'm supposed to be working on)...

Perfomance Issue

When I do a backup, restore, or worse (copy a backup from 1 vm to another to move it to the QA VM), the entire system slows to a crawl (even "unrelated" VMs).

I originally thought a DASD situation would be quite good since you had PCI-x bandwidth, but the systemwide slowdown is killing productivity.

Questions

  1. What should I do to make an intelligent decision about NAS vs RAID vs SAN vs DASD?
  2. Are there sweet spots/ugly spots in the storage setup?
  3. Can you use a SSD PCI-X card in ESX for the tempdb? Good/Bad idea?
  4. Is there any way to "share" some image in a copy-on-write fashion? Most of the "Backup-Copy-Restore" is to "put a clean image on the dev boxes"; if I could have them "share" the master image, the "big copy" (2x50 GB) would only need to be done once per week instead of once per dev per week...[runtime performance isn't a concern with the dev boxes, but the backup/copy/restore kills production, SBS, and everything else on the box]
share|improve this question

4 Answers 4

up vote 1 down vote accepted

You don't provide details on the ESX server specification - number of CPU's, amount of RAM, how many VM's you are running and what their configuration and expected performance requirements are so I'm assuming all of those are OK for the moment and that the problem you have is definitely just storage IO related. At a minimum you should have sufficient cores and RAM to run all of your active VM's with no contention and enough headroom to ensure there is sufficient capacity for the hypervisor to do everything it needs to do without halting VM's or forcing them to page out to disk. Also as Frederic mentioned you must have VMware Tools installed, and if you can use the paravirtualized SCSI adapter for your data drives that will help.

Question [1] - DAS\RAID\NAS\SAN

You need to provide more IO and more independent IO to separate workloads from each other. Whether you go for NAS\DAS\SAN this will require setting up separate RAID packs, preferably in RAID 10. For your style of IO the performance benefits of RAID 10 over RAID 5 (double the write IO performance for the same number of fisks) is advisable. You want to ensure that the primary IO intensive VM is running from a separate datastore to the QA system, especially if the QA procedures are likely to seriously stress the storage which I assume it will. Breaking your current 8 drive pack down into 2 separate RAID 10 packs of 4 would help a lot. Changing the existing 8 drive RAID 5 pack to RAID 10 and adding separate independent storage for the QA system would be better. You can do all of this with direct attached storage but performance will scale better with SAN's and high end NAS systems. Those will also cost a lot more though.

Question [2] Sweet spots/ugly spots with existing storage

You have enough disks to run about 10-15 average Windows servers (average being file\print\DC's\low IO app boxes not SQL, SBS or Exchange) but not enough to run much more than one truly IO intensive machine in the current set up. The ugliness is having a single RAID 5 pack for everything. If your existing drives are SATA then that's not good as they are likely to be 7200rpm at best, moving to 10kSAS drives will more or less double performance for the same spindle count, and 15k drives will triple it.

Question[3] Using PCI-X SSDs

ESX doesn't support and PCI-X\PCI-e SSD's AFAIK but if it does then that would help as a location for any of you IO intense VM volumes. As far as running tempDB is concerned then ideally, yes, you should put it on a VMDK that sits on fast storage but you should first make sure that the VM has as much RAM as it can profitably make use of.

Question [4] Shared Image Options

VMware Lab Manager will do this sort of thing for you but it requires a fully licensed VI infrastructure and a per CPU license for each server in your cluster. It will help with management and has some potential positive's for your QA copy use case but it is also possible that this wont help resolve performance issues - if the QA system is being driven hard from an IO perspective then the Copy-On-Write\delta copy approach used to provision multiple systems from single base image does not help your general performance problems and will probably make them worse. It will make creating the copies much easier and faster though.

share|improve this answer
    
THere are wonderfull 10k SATA drives from Western Digital (Velociraptor) - a lot cheaper than SAS drives. And a LOT faster than 7200 RPM drives. 2.5" factor (get the enterprise version). –  TomTom Jun 5 '10 at 17:57
    
Good point - they are excellent enterprise grade SATA disks that are much better than 7.2k disks if you're in a position to use them. –  Helvick Jun 5 '10 at 19:07
    
What would you recommend as s SAN/NAS? I'm willing to throw some money at the problem, especially if it will improve things (and since VMotion, HA, et al aren't supported with a DAS, it seems I chose poorly with a DAS system). What's the best way to get out of the current box? I'm thinking about: –  user45006 Jun 6 '10 at 13:26

Your problem is not related with DAS or NAS but with CRAP DISCS. A lot of VM's are a lot more IO than a normal computer. Usage of FAST discs is mandatory here unless you really have low usage patterns.

  • You are ok for a smaller setup with a DAS system, if you have a good RAID controller (Adaptec line) and a good number of discs.
  • SAN (not NAS) has the advantage of being shareable between multiple boxes (failover, clustering etc.)

Whatever, though, you need a LOT of possible IO. Rules are:

  • VM's should NOT USE THE PAGE FILE. Paging stresses disc - and discs are your brutal limtiation anyway.
  • RAID 5 is a lot slower than RAID 10 ;)
  • 500gb Discs are most likely 7200 RPM maximum - and that is slow. Go 10.000 RPM minimum, SAS infrastructure. I have really good experiences with a RAID 10 of 300gb Velociraptors (though the new 600gb are a little faster).

You really need a LOT of IO capacity. Your RAID 5 is only marginally faster than a two discs (if at all - I would have to make the math) and a single disc is already not exactly fast for certain scenarios. THe result is - that your IO setup just dies. Normal.

If you would install a RAID 10 of 8 discs (500gb wise that is 2tb) you would get a LOT more independant IO capacity than with a RAID 5.

share|improve this answer
    
The RAID 5 pack is going to have ~1.8x single drive performance for writes but should be close to 6x for reads. Doing a copy from:to datastores on the same system would probably run at somewhere between the speeds of one and two disks though so your not far off. RAID10 will help a bit, it will double write performance but getting better disks and more spindles is probably the best first step here as the load seems to require it. –  Helvick Jun 5 '10 at 14:53
1  
Let's not forget that raid 10 is almost every time much faster then raid 5 in IOPS. If you have a lot of concurrent access to the storage, raid 10 is (almost all the time) much better then raid 5 –  Pier Jun 5 '10 at 14:56
    
This is wrong- helwick. ARaid 10 allows a LOT more independant writes in larger arrays. A LOT more. Especially with larger discs sets. –  TomTom Jun 5 '10 at 17:56
    
@TomTom - I beg to differ - the write penalty for RAID5 is 4 (two disk reads and two disk writes are needed for every write IO). The write penalty for RAID10 is 2 (two disk writes must be made for every write IO). RAID 10 can handle twice the write IO levels that RAID 5 can before saturating. For Reads they are both the same provided your controller is reasonably good. RAID 10 is much better certainly but it does not support more than double the write performance of RAID5. –  Helvick Jun 5 '10 at 19:35
    
Increasing IOPS from, let's say, 1000 to 2000 (impossible, probably around 1600) just changing type of array, it's not so bad ;) Important is to consider the ratio of read/write operations. –  Pier Jun 5 '10 at 21:27
  • (1) and (2) : if you use a local storage, you won't be able to use all the vmware features (ha, ft, vmotion, etc.). I use iSCSI on an EQL SAN : performance are really poor, due to a bad iSCSI implementation (around 2Gbps in write) and the raid6 used. With raid10 IO performance should be really better. So, if you have money, use a FC SAN... or wait for FCoE SAN :) with raid10... In my opinion, for intense IO, don't use low cost SAN...

  • (3) I don't know

Have you tried to set up the guest tools in order to reduce your slowdown ? We had the same problem with an Oracle database and setting up the guest tools really increased performance. We also used the paravirtualized scsi drivers in our RHEL to gain more performances !

Frederic

share|improve this answer
    
Even 2gbit are a LOT of io if you hit the SAN side. THis is roughly 200mb / second - a 8 to 12 disc RAID 10 will have problems giving you that all the time ;) –  TomTom Jun 5 '10 at 17:58
    
I wasn't aware that Equallogic supported RAID 6, I thought they only did 5\10\50. An EQL environment's bandwidth scales out pretty linearly at a rate that's not far off 450MBytes/sec per array (sequential read on the PS6000 series) - I've just put in a set of 4 that easily handled a 1Gbyte/sec read test without impacting 10 test VM's running on the same group. SAN performance is about a lot more than raw sequential transfer rates but there's not much to complain about with EQL when it's done well. –  Helvick Jun 5 '10 at 21:12
    
That's what I found when I moved to ESX 4.0 from VMWare server. It seems that Local Storage has been phased out and I didn't get the memo somewhere. I'm using the Paravirtualized SCSI adapter on all they nodes... is there another thing that I need to enable on the tools? –  user45006 Jun 6 '10 at 13:18

Could be this your problem?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011213

Anyway update all the bioses. Are the disk sata or sas? Judging by the space I'd say sata...not the fastest disks. If you want the best performance, you should try raid 10 instead of 5. NAS, SAN and DAS are one thing...RAID is the base for almost all of them. Shortly:

NAS: export a file system (nfs, smb)
SAN: export a block device (iscsi, lun, aoe)
DAS: all the storage boxes connected directly to the server (scsi, esata)

RAID: is a way to increas availability of the system in the case of some disk crash.

share|improve this answer
    
That sounds like it might be the issue; they're 500GB Seagate disks, but I'm not sure if they're SAS or SATA with an interposer. I'll see of that firmware helps. –  user45006 Jun 6 '10 at 13:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.