0

I have a EqualLogic PS4000 SAN unit with the latest firmware, setup in RAID 10. I have 3 2TB Volumes on the SAN shared out via iSCSI on 2 eth ports on two different subnets. I have moved a test server over to this newly setup SAN, and my testing is showing me a problem.

I am getting dismal read performance in everything except a test with 32 queue depth (see attach image)

enter image description here

Write performance seems to be right about where it should be.

I have tried MPIO on and off, on was slightly better but not much.

Improve this question
7
  • Are benchmarks running one after another, or at the same time?
    – nanofarad
    Aug 24, 2014 at 23:49
  • One after another on a single test VM.
    – Litzner
    Aug 25, 2014 at 0:21
  • Not sure if the PS4000 has a Read-Ahead setting, if so, disable that since virtualization reads are random and not sequential.
    – Reality Extractor
    Aug 25, 2014 at 2:41
  • I'd try a different tester with a more controlled set of test conditions. Doing such testing without planning carefully can lead to incorrect or misleading results. Also remember iSCSI is ethernet based and as such external traffic can drastically skew results.
    – mdpc
    Aug 25, 2014 at 3:08
  • 1
    @mdpc Please get real. Ehernet will not skew results to a degree like this. One can argue whether 82MB are limit or not for even 1gigabit ethernet, but 4 MB(!) are so slow that a DSL line would be able to handle it. UNless the link is seriously overloaded it is comically low.
    – TomTom
    Aug 25, 2014 at 5:12

2 Answers 2

Reset to default
3

The first thing to note is that you're not actually testing the SAN performance here. Since your IO benchmark is running on the C drive of the virtual machine, which I would guess is a virtual hard drive stored within the file system of one of those 2TB volumes (e.g. a VMDK stored within a VMware VMFS datastore). You have a lot of added layers to the I/O path that you should not have for testing purposes if this is the only test you've run on this volume, or even on this VM.

I would recommend that you give your virtual machine direct iSCSI access to a separate test volume, format the volume, and run the test again on the test volume.

Secondly, you need to determine whether or not the IOPS load on the disks is a factor in your results, assuming that this array is already in production and has a regular workload on it. SAN Headquarters is provided by Dell at no added cost, provided that you have an active warranty on the system. SAN HQ gives you data about IOPS on a per-spindle basis, and can show you whether or not I/O is queuing up badly when you're running these tests.

With 16 x 7.2K spindles and the MPIO/network setup that you have, you should easily be able to saturate the single gigabit link that this VM has available to it (due to your split-horizon config, which I address below). If any of that single link's bandwidth is being used by other I/O, that's another factor that will limit or potentially interrupt your results.

shared out via iSCSI on 2 eth ports on two different subnets

This will definitely contribute to performance issues - Equallogic arrays are not designed to be used with multiple iSCSI subnets, and this configuration is not supported at all. With your current configuration, you have no network-level redundancy on the EQL array side (if a network link goes down on the EQL side, one subnet loses all iSCSI access).

The last factor to mention is the Hypervisor itself. It's possible that issues with the physical host configuration or hardware might be a factor as well. If you're able to completely rule out disk IOPS workload and network bandwidth availability as the culprits of your performance problem, you may need to seek assistance from a support provider. I would highly recommend contacting Dell's Equallogic support team, especially if you're using VMware ESX.

Improve this answer
6
  • Note: Using a PS6010E array (10Gbps networking), I'm able to pull down 250MB/s on Sequential Read using the exact same CrystalDiskMark settings... And that's with RAID50, not RAID10. So the 7.2K spindles are definitely able to give you much better performance on sequential reads than what you're getting in your tests.
    – JimNim
    Aug 25, 2014 at 15:33
  • Thanks for the tip, I am going to give a configuration of the same subnet for each port a shot. I had always set each port on separate subnets due to routing issues that would prevent MPIO to work correctly before, but that was on different SAN units, where maybe the routing table in this SAN will not have that issue.
    – Litzner
    Aug 25, 2014 at 18:17
  • I have modified the configuration to put both eth interfaces on the same subnet, and now both Read and Write are preforming worse then they were before. The read performance down about another 10%, and the write performance down 30-40%.
    – Litzner
    Aug 26, 2014 at 14:37
  • I just had a thought... My PS4000 is reporting the eth links operation at 9000 MTU, my switches and networking are setup for 1500 MTU. (When ever I have attempted to use jumbo frames in the past it had issues) Could this be causing my issues? I have attempted to edit the MTU on the PS4000, but it doesn't appear to be a changeable setting.
    – Litzner
    Aug 26, 2014 at 14:47
  • I have enable 9000MTU (Jumbo Frames) on everything through to the PS4000 (physical switches, vswitch, and created vnics just for the PS4000 with 9000MTU), and I saw a slight performance bump, but still terrible read performance.
    – Litzner
    Aug 26, 2014 at 15:05
0

I have found the issue!

The problem was in the networking layer between my hosts in the SAN. I first directly connect the SAN to a host and I did not have a problem. So I hooked it back up the way it was and started troubleshooting the networking. The traffic that this particular PS4000 SAN puts out is apparently adversely affected by Spanning Tree Protocol. Once I disabled STP on the ports associated with the PS4000 SAN everything started to work as it was supposed to.

Thank you for all the input given to me that helped lead me to this.

Improve this answer
1
  • Glad you got things figured out! In the future, keep in mind that most switches "validated" for use with EQL storage actually have switch configuration guides available as well - tough to go wrong when you follow those guides too.
    – JimNim
    Sep 26, 2014 at 21:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .