4

I have a performance issue with my SAN. I have a full 10Gbps private network setup. I have the following setup

1 - Mac with Connected with via Fiber Transciever 10GbE
1 - Windows 7 PC with 10GbE
1 - HP Switch with All ports are 10GbE
2 - Quantum StorNext Servers with 10GbE
1 - Dell Compellent Solution with 2 Controllers connected to the network at 10GbE
All servers, switches, and computers have staticly assigned IPs

Compellent Config
Tier 1 - 10K SAS Drives in RAID 10
Tier 2 - 7200 SAS Drives in RAID6 (12 groups of 13 Drives in Each)

Additional info
Windows 7 Client TCP Offload Options
IPv4 Checksum Offload - Enabled
TCP Checksum Offload - Enabled
UDP Checksum Offload - Enabled
Large Send Offload - Enable
Jumbo Packet - 9014Bytes Enabled

I mounted the StorNext volume on my Windows 7 PC and my Lion Workstation. Unfortunately all my transfer speeds are at around 2Gbps or 2.8Gbps (if I'm really lucky). I was looking to get at least 5Gbps speeds out of this setup, but I'm averaging about 2Gbps or a little above 250MBps transfer rates on file copying. When I map a LUN directly to the boxes and then format it natively with either HFS (with journaling) on the MAC or NTFS on the Windows 7 PC, and then copy a file, I get about 180MBps. So my performance on a directly mapped LUN is slower than my StorNext Volume. Any suggestions? Has anyone seen degraded performance on iSCSI with 10GbE? Any help would be Awesome! Thanks!

9
  • 5
    This is probably a dumb question but is your storage system capable of transfering more than 2.8 Gbps? Jul 23, 2012 at 23:24
  • 2
    So many possibilities... have you looked at CPU utilization? MTU size? TCP Offload settings? Packet loss? NIC driver issues? Jul 23, 2012 at 23:24
  • 1
    What are the type (SAS/SATA), size, rotational speed of the drives? how many of them are there and what RAID configuration are you running them in? Jul 23, 2012 at 23:48
  • I have a Tier 1 Filled with 5TB of 10K SAS spinning drives in RAID 10. My Tier 2 is all 7200rpm drives in 12 groups of 13 drives in each All in RAID6. I haven't measured packet loss. My MTU Sizes are the standard 1500bytes. CPU Utilization is really low. CPU on my StorNext servers are really low, like around 15%. My Compellent Doesn't give me CPU metrics, but I'm hitting a max of 2,000 IOPs with IOMeter. The Max speed we were able to hit briefly with IOMeter was 600MBps, but that wasn't fully sustainable. Not sure what my TCP Offload settings are either.
    – cohortq
    Jul 24, 2012 at 0:05
  • I think you're hitting packet per second limits. A larger MTU will likely help. (Is your MTU 1500 or 9014?) Jul 24, 2012 at 0:24

4 Answers 4

4

1.) Jumbos -might- help if you're seeing a lot of processor load for interrupt traffic but if TCP is operating correctly it should be able to ramp well past 2G on a 10G link. I've seen plenty of 10GE links running above 90% without jumbos enabled.

2.) If you do use jumbos, enable the same size on every NIC and every switchport in the VLAN and/or broadcast domain. PMTU works when packets cross routers and mixing MTU values within the same network will lead to nothing but misery.

3.) I'm not particularly familiar with the Procurve gear but TCP traffic can be tricky at high speeds if there are any questions about buffer availability. I've seen other testing where this has manifested (without apparent TCP drops) as a huge cut in performance that ended up being fixed by actually reducing buffer sizes.

4.) Make sure that the actual TCP settings (1323, SACK, etc) are all configured consistently. The operating systems in question should be fine out of the box but I don't know much about the storage node. It might be worth digging into - either in terms of settings on the device or via a protocol trace (wireshark or tcpdump) to observe window sizing and any retransmissions going on.

5.) Try eliminating as many variables as you can - even getting down to a crossover cable between one of your storage nodes and a single workstation - to further isolate the issue. Don't be afraid to disable some of the offloads you mentioned as well, as they've been known to cause issues from time to time.

1

If you're using 1500 MTU, piece of advice #1, use jumbo frames/packets at ~9000. It does seem you have Jumbo packets enabled to some extent (based on "Jumbo Packet - 9014Bytes Enabled") so it is also possible you're dealing with packets getting fragmented as well. Make sure everything along your path is set to allow the 9014 byte packets.

1

Not the answer you'll be looking for, but our Compellent SAN has been nothing but a disappointment performance wise. I too had similar performance numbers (Before I get jumped on... this is on a dedicated 10G iSCSI switch using Jumbo frames, and everything was validated by both CoPilot and Dell's "Tiger Team". Our other SANS (EMC and many Equallogics) have comparably blistering performance). What got us out of this particular performance bottleneck was replacing a faulty controller.

Have you had co-pilot look over the configuration? If properly set up, the SAN will be reporting home anyway, and they may already have it flagged.

What interface speeds do you have on those client machines? Any interesting network setups? Up to date drivers? What does CPU/Memory look like in Resource Monitor while doing a transfer test?

1
  • I have everything connected on a separate 10GbE network. I have 2 Controllers and they are each connected to a dedicated HP ProCurve on 10GbE. How was your Compellent configured? What do you think was the main bottleneck in performance after the Controller replacement?
    – cohortq
    Jul 24, 2012 at 20:18
0

If you're worried about network performance try iperf to test network speeds between computers; this would help rule out network related issues.

Have you considered disabling jumbo frames altogether?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .