1

Read performance on our SAN is slow under certain workloads. When we compare this to some local storage, we find the local storage performing 2x as fast. The SAN performs well with a high Queue Depth, and poorly with a low queue depth. However, the local storage performs well with a low Queue Depth. I'd like to know the reason for this occurring and find out what the specific limiting factor is in this situation.

MD3200i iSCSI SAN ($15,000)

  • 6 x 600GB 15k SAS RAID5
  • 6 x 2TB 7.2k NLS RAID5

XCOPY /j Benchmark: (Slow)

  • 15k Array - 71MB/s (Queue Depth 1)
  • 7.2k Array- 71MB/s (Queue Depth 1)

Robycopy /MT:32 Benchmark: (Fast)

  • 15k Array - 171MB/s (Queue Depth ~12)
  • 7.2k Array- 128MB/s (Queue Depth ~12) ,

, Read Performance on a Local controller is fast under the workload the SAN is slow at.

,

HighPoint 2230 RAID Controller ($600)

  • 4 x 1TB 7.2k SATA RAID5

XCOPY /j Benchmark:

  • 7.2k Array - 145MB/s (Queue Depth 1) (appears to max out the SATA bus)
Improve this question
1
  • We get 30 MB/s with a 10 GB volume on an otherwise empty RAID 0 disk set consisting of 3 2T SAS drives. I tested this with Iometer and with copying files. Yeah, I'm not impressed. Something's most likely wrong, and I'd love to hear what others know about this. I'm using only one 1 Gb connection right now but it maxes out at 33% and I have no clue why. I get the same transfer limit when writing to other volumes on the same SAN (different RAID set). If anyone has any ideas as to what could cause that, please let me know because system specs can't be the issue: a dual Xeon E5620 blade with Win 20
    – Jacob Bruinsma
    Apr 6, 2012 at 22:44

1 Answer 1

Reset to default
3

You don't mention how the networking for iSCSI is setup but if I had to guess you'll only be getting a maximum of 1Gbps of theoretical bandwidth right? So that's ~100-125MBps MAX, on top of that you have both the IP protocol and the SCSI protocol - both of which eat up both bandwidth and latency - so you're not actually doing too badly although I'd have expected more like 80MBps. Now compare that to the 300MBps minimum bandwidth you'll be seeing via SATA and PCIe - there's SO much more bandwidth and virtually no protocol translation, even then it'll be done as much lower latencies.

I think that's you issue, hope it helps.

Improve this answer
4
  • the iSCSI is currently setup with 2 nics functioning in a round robin setup. Theoretical would be 250MB/s. Additionally I tested up to 4 nics (500MB/s theoretical) with the same result. In both cases, I get 171MB/s when the queue depth is high, so that tells me the bandwidth is there.
    – Caleb_S
    Feb 9, 2012 at 23:03
  • It just seems hard for me to believe that the iSCSI layer adds 70MB/s of overhead for certain workloads. I guess it's one one of three things. 1. iSCSI Hardware Layer is Slow 2. iSCSI Drivers are are slow for this type of workload. 3. Both
    – Caleb_S
    Feb 9, 2012 at 23:32
  • I guess the one last thing is that it could be the way the RAID controller handles single que depth I/O. solidq.com/sqj/JournalDocuments/2011_January_Issue/… on page 3of8 in this article in the section titled "RAID Group Random Read IOPS versus Queue Depth", it talks a little bit about single queue depth i/o and how the raid controller will only distribute it to one disk.
    – Caleb_S
    Feb 9, 2012 at 23:41
  • After some more testing, I have come to the conclusion that the iSCSI/tcpip protocol is where the performance hit comes in on this particular type of workload. The final test would be to swap this unit with a direct atttach SAS model. We'll see if that happens. Thanks.
    – Caleb_S
    Feb 16, 2012 at 21:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .