I have a VMware ESXi 5 environment running on Fujitsu BX922S2 blade servers, connecting to a Fujitsu DX90 SAN.

I've had no problem with RAID5 volumes, but RAID10 volumes (for MSSQL) are exhibiting ridiculous write latency - up over 1000 ms at times, and never lower than about 30 ms. These latency figures are being measured in VMware, using the Performance tab for the SQL server concerned, changing to Advanced, then switching to Datastore view.

The DBA noticed this, and wrote a quick script to continuously create a 1GB file on the server.

Migrating the SQL logs drive to a RAID 5 volume on the same SAN shows no perceptible write latency.

Can anyone point to anything obvious I've configured incorrectly that would cause this?

  • can you provide more info about your RAID 10 vs RAID 5 config? Meaning number of spindles, is it on a different san controller, types of drives, etc. Seems kind of odd that latency would be higher on a RAID 10 compared to 5 if everything is equal. Jan 16 '12 at 1:55
  • Thanks Eric, and yes, RAID 10 should perform better, that's why it's been configured that way. The RAID 5 and RAID 10 volumes are both running off the same controller in this case. The RAID 10 is the minimum it can be in terms of spindles (4). The RAID 5 volume is across 12 spindles. The RAID 10 is only being accessed by two servers; the RAID 5 contains all the c:/ drives for multiple VMs.
    – slag
    Jan 16 '12 at 3:28
  • How are your drives laid out in the RAID10? Are they part of the same shelf or are they spread out across two or more shelves? Can you controller have custom cache optimazation per LUN? I'm just wondering if your shelf is overwhelemed, or if its specific to a particular shelf (if the drives are spread). BTW, I presume your SAN can report performance of a given volume, is there anything going on for that LUN or shelf? Can you carve it differently? I would try moving it to a completely different shelf just to test it. Also, what's the current throughput on your shelf with the RAID 5? Jan 17 '12 at 0:34
  • oh, and what about trying just a simple RAID 1, not 1 + 0 Jan 17 '12 at 0:34
  • The RAID10 volume is over two shelves (the controller shelf and a disk shelf). There are some very basic cache settings, which can be altered per LUN - currently talking to the hardware vendor about these. I've now created separate RAID0, RAID1, and RAID5 volumes, and all seem to perform equally poorly. The original RAID5 still performs perfectly.
    – slag
    Jan 19 '12 at 19:49

My SAN experience is with virtual SANs, but to me it sounds like you're down to the following.

  1. You've indentified that the current RAID 5 works fine, but no combination of RAID 0,1 or 5 is working well. You clarified that orignally you were doing a RAID 10 acorss two shleves and that you've tried other simple configurations. What you left out is if you're once again trying these configurations across shelves? I would try a simply RAID 0 on one shelf, and then another one on another shelf. Perhaps its just one shelf that's overwhelmed.
  2. You don't mention if you've tried this with a non-vmware host. One thing I would try is connecting via something else that's outside. This is simply to eliminate a VMware configuration issue.
  3. How does your switching look? Do you have redudant switches, do you have multiple HBA or at least a dual port HBA?
  4. I presume its not a network issue (san wise) only because i would think you'd see a fluxuation in peformance across multiple luns
  5. Do you have multi pathing enabled in vmware? I suspect fujitsu has an expension pack for this.
  6. If it was me, I would call the SAN vendor and ask why it might be slow. Perhaps there's something obvious to them.
  7. BTW, i assume this is all fiber channel?
  • Thanks for the comprehensive answer! 1. All the new volumes created for testing have their disks on one shelf each, i.e. no one volume has disks across multiple shelves. 2. Haven't tried this with a non-VMware host, will have to do so. 3. All servers are blades, the chassis (2 separate chassis for the servers involved) have dual Brocade switches & the servers have dual HBAs 4. My thoughts exactly! 5. Will look into this. 6. Currently talking with Fujitsu, I too hoped there was a stupidly obvious bit of config I'd missed. 7. Yes.
    – slag
    Jan 23 '12 at 22:39

Figure out whether you can pin the write latency issues down to a specific controller. Check all your cabling. Check your FC/iSCSI switch for errors on a specific port. Try testing without MPIO enabled. Try testing writing directly to the volume outside of VMware.

After that, make the manufacturer figure it out.. this is definitely their problem to solve.

  • Thanks for the reply! The SAN has dual controllers, have tried swapping each volume between them, no discernible difference. Cabling between servers & Brocade switches is ruled out as we're dealing with internal wiring in a chassis, and it doesn't appear to matter which of two different chassis I use. Currently looking into MPIO to see if that makes a difference. Also gathering data from all the test volumes to send to Fujitsu.
    – slag
    Jan 23 '12 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.