3

I have a brand new Dell VRTX box set up as a Failover Cluster running HA Hyper-V virtual machines. This is my first time setting up clustering, and my first time with one of these boxes, so I'm sure I've missed something.

The virtual machines are experiencing high disk latency and bad performance when accessing their VHD(x) files located on a Cluster Shared Volume.

The VRTX has 10 x 900 GB 10K SAS drives in RAID 6 configuration, and the VRTX has the redundant Shared PERC 8 controllers. Both blades have full access to the virtual disks. There are two M520 blades installed, each with 128 GB RAM. MPIO is configured for the PERC 8 controllers. Operating system on the blades is Server 2012 (NOT R2).

The RAID 6 array is split into a small (8 GB) volume for cluster quorum witness and a large (6.5 TB) volume for a Cluster Shared Volume (mounted on the nodes as C:\ClusterStorage\Volume1)

An example of slow disk access: logging into a Server 2012 VM and having Server Manager come up automatically. Disk access goes to 100%, with write speeds at 20 MB or so, read speeds of 500 KB or so, and Average Response Time of over 1000 ms, sometimes spiking at 4000-5000 ms or so. It's the latency that really worries me.

Is there something specific I should look at in my configuration? It doesn't seem to matter whether I use VHD or VHDX, dynamic or static.

3
  • Have you actually formally benchmarked it? – Chopper3 Jun 2 '14 at 17:02
  • @Chopper3 - Well, not really. I tried running a few different disk benchmark programs on the blades themselves, but couldn't figure out how to get them to benchmark a volume mounted as a directory, and their write tests were destructive. I bet I could run one of them on a VM though, with a new VHD. I'll try that. – NorbyTheGeek Jun 2 '14 at 17:07
  • You could try using IOmeter, as it's able to create a test file w/ a size you choose (specified by # of sectors, 512 bytes each for that system). I'd also recommend checking the latency that's reported in PerfMon if you're looking at the data in TaskManager right now - I don't know what data its graphs are based off of, but I personally wouldn't trust it alone. Also, if this is a brand new unit, don't be afraid to call support. They've likely seen this problem before, and would be happy to help work out the kinks. – JimNim Jun 2 '14 at 18:26
6

I have experienced the exact same performance issue with a VRTX with the Dual SPERC8. What I have done to work around this at the moment is, change the dual config to a single config. This way I am able to use write-back, which performs way way better.

  • The exact steps:
    • Remove the second SPERC 8 controller
    • Remove the second expander
    • Re-cable the internal SAS connections
    • Downgrade chassis to 1.25 (works the same as upgrading, no special steps needed)
    • Delete all VD's (backup data/VM's if needed)
    • Power cycle entire VRTX (remove and reconnect power cables to be sure)
    • create VD('s) with write-back enabled

To see the performance difference check this/my thread at: http://en.community.dell.com/support-forums/servers/f/906/t/19587459.aspx

Update:
Test results:
Dual PERC / RAID6 / Write Through: Read 2500 MB/s Write 200 MB/s
Dual PERC / RAID10 / Write Through: Read 2500 MB/s Write 400 MB/s
Single PERC / RAID6 / Write Back: Read 2500 MB/s Write 2700 MB/s

As long Dual PERC is bound to Write Through policy, I would stick with a Single PERC setup

5
  • Thanks for adding specific performance data, although it typically would not be about plain throughput but the number of IOPS you can do with the setup. The most important and most revealing sentence from the Dell Support reply to your Dell Forums post is "The VRTX is not a full-featured storage appliance". They really should post that prominently to the product page and into tech specs. – the-wabbit Jun 11 '14 at 9:47
  • @syneticon-dj I have contacted support and they recognize/are aware of this performance issue. The specialists, so they say, are looking into this issue. I am currently waiting for a troubleshooting and performance tuning guide for the Dual Controller setup. – Erik Jun 12 '14 at 9:40
  • This rather seems like a design decision than a support issue. I am curious about how they are going to "fix" it, assuming the hardware as well as the firmware are coming from LSI. – the-wabbit Jun 12 '14 at 10:09
  • I received an official reply by Dell. They admit that a Dual PERC setup is not suitable for any virtual environment. They also recognize that Dell sales department wasn't correctly trained on this configuration. So @NorbyTheGeek maybe you need to work something out with Dell as well. See my earlier mentioned thread for some more details. – Erik Jun 15 '14 at 20:46
  • I have been following your Dell thread, and this unfortunately is the answer. The process went fine, I did not need to delete the VD's, just downgrade / upgrade the CMC firmware. With the single SPERC8, write caching is enabled and performance is greatly improved. – NorbyTheGeek Jun 18 '14 at 16:35
7

This:

Fault Tolerant Shared PERC 8 Card Configuration — [...] The default cache policy for virtual disks created in this configuration is write-through. In this mode, write completion information is returned to the host after the data is written to the disk.

is the ultimate performance killer. Change the cache policy to write-back if it is supported for your application and does not result in possible inconsistencies within the written data. Note that I have no idea if and by which mechanism the PERC8 cache is being mirrored to the other instance. As the cached data needs to be accessible from both controllers, it obviously would be necessary for consistency.

6
  • It would appear that write-through is the only method supported. I attempted to create an additional virtual disk and the only choice it gave me was write-through. – NorbyTheGeek Jun 2 '14 at 20:05
  • @NorbyTheGeek try triggering Dell support about this. Virtualization loads on non-writecached controllers with HDD arrays are known to perform poorly due to a high amount of synchronous writes they incur. – the-wabbit Jun 3 '14 at 5:23
  • If you actually did turn on local caching in the controllers, that would invalidate the cluster file system that you're trying to use, which is probably why you're not seeing the option to do it. If you want to use CSV, any caching has to be coherent across all the hosts that have access to the volume. – Jake Oshins Jun 3 '14 at 18:10
  • @JakeOshins there are cache mirroring mechanisms, mostly seen implemented in dual-controller storage appliances. I have no idea if the PERC8 has any support for cache mirroring. – the-wabbit Jun 3 '14 at 21:21
  • Yes, that's why I made a distinction between local caching and coherent caching. – Jake Oshins Jun 4 '14 at 16:36
5

It is no longer required to remove the second SPERC controller to be able to use write-back instead of write-through as described in Erik's post. You can now disable the second PERC8 controller from the CMC. In the current firmware (1.35), the second disabled controller can be set as disabled and requires manual intervention to activate if the primary active controller were to fail. The usefulness of this is not really detailed in the patch notes, but this "fix" is meant to allow people to enable Write-Back and get rid of the abysmal performance you have when using both controllers in Write-Through.

Automatic failover (cold failover, causing disconnect) is a feature that will be release later. A lot later will be the actual "active/active" firmware update, which would allow "live" failover without downtime.

Steps:

  1. Download VRTX CMC firmware version 1.35 or higher.
  2. Shut down all your blades.
  3. In the CMC interface, press "Chassis Overview > Update".
  4. Select both checkboxes for the CMC controllers in the "CMC firmware" heading and press "Apply CMC Update".
  5. Enter the location of the CMC firmware file and Apply it.
  6. The CMC will show you its progress. Uploading takes about 8 minutes, applying the update a few minutes more. The CMCs will reboot after applying the update and you will get kicked out of the webinterface.
  7. Once the CMC has rebooted, browse to the "Storage > Controllers > Troubleshooting" section.
  8. For the SPERC of your choice, select the "Disable RAID controller" option and apply. This will reboot your storage component.
  9. Once rebooted, go to "Storage > Virtual Disks > Manage" and select "Edit: Write Policy" and pick "Write Back" instead of "Write Through" for all your virtual disks (unless you have a reason not to).
  10. The change above will be carried out immediately, but it is still suggested to reset the CMC once more under "Chassis Overview > Power > Control" with "Reset CMC (warm boot)".
  11. Boot your shut down blades.

This will allow you to have a second PERC8 installed in your VRTX, in case the other fails. But you will have to do a manual intervention to fail over in case of failure. I suppose this is primarily meant for hard to reach locations (remote offices without IT staff or easy access for Dell support technicians). This is also what we use it for.

Hopefully by end of the year we will have the automatic failover feature and then in the course of next year the true active/active config with write-back enabled (synchronized caches). I'm not going to hold my breath for the synchronized cache firmware fix... I suspect that won't be an easy one for Dell.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.