iSCSI MPIO load balancing with HP P2000 SAN

Question

My setup:

A hyper-v cluster with 2 nodes (AMD 24 cores, 64gb ram) connect via 2 procurve 25-10g to a p2000g3 SAN (12x300 GB enterprise SAS 10k 2.5"). Each node is an iSCSI initiator.

Problem:

When I run a VM on a node and launch a basic crystamark disk benchmark, my MB/s are 80 for both read and write. This is poor.

What do you guys think ?

Update1: After a long day of testing here is my first conclusion. It seems i can't expect more than the bandwith of one gigabyte ethernet card. When no mpio and so only one connection, the bandwith is used at 100%, when 2 mpio paths, each path takes 50%, when 3 mpio paths 33%... Enabling flow control does nothing more ( less it seems..), no more for enabling jumbo frames. I can't imagine i'm stuck to 1gb, the p2000 has 2 controllers with 4 ports each. Please tell me it is possible to aggregate this :)

For recap my tests are: Running a crystalmark disk sequential (1000mb) on a vm give me between 80 and 100 mb/s, the same test with the storage vm locally on the hyperv host gives 250.

Answer 1

We had a very similar setup a few years ago. It's a complex setup with lots of moving parts that you have to look through. Here are the main areas to check for performance:

• SAN: Make sure it's configured properly, verify your virtual disk layouts against the manufacturer specs, virtual disks that are spread out often cause problems. In addition make sure you have the proper settings to deal with two or more hosts (cluster) reading from the same virtual disks.
• Switches: Check your jumbo frames settings, you typically want to make sure jumbo frames are enabled on your switches, SAN and servers for data connections only. Also make sure you are using a VLAN to segregate the traffic if both your network and data traffic share the same switches.
• Server: Drivers and iSCSI, make sure they are up to date and setup properly. With multiple hosts you really want to use multipath drivers and configuration, again check with your manufacturer. You didn't post your NIC specs but make sure it supports 1GB or higher traffic as well.

That's some real basic starting points. Worst case call up your vendors, they typically have good white papers or case studies around this kind of stuff.

Answer 2

80MBps = 1Gbps Ethernet, you are either only using 1 NIC or you don't have MPIO setup correctly.

Answer 3

Your performance will vary based on the type of benchmark. If you're doing sequential reads or writes, you should be able to get a large amount of MB/s performance from the disks. This is not a useful benchmark because that's not the type of IO a virtual server does, usually.

If you're doing small block random IO, your writes will be cached until you run out of cache, and only a fraction of your reads will come from cache. The rest will all go to disk, and then your performance in terms of MB/s is irrelevant. The number of IO/s will be limiting you. Each 10k disk can add as many as a couple hundred IO/s, so with 12 of them, you can't get more than a few thousand.

edit: If you're using a 1Gb/s network switch, then the maximum throughput you can get is 80MB/s, which is right about what you're seeing. Add network ports on the switch and see if it improves.

Answer 4

if you have 2x1gig connections per server and 2x1gig connections per SAN node, then you have MPIO misconfigured on the server, or transmit load balancing on the SAN.

if you SAN does not support transmit load balancing to a single host then you're out of luck: the most you'll be able to get TO a host is a single link.

i can't find any documentation that the p2000 support transmit load balancing; i only find references to active/passive controllers.