3

We just purchased a dell storage server with (12) 7200 RPM SATA drives running RAID 10 and (4) gigabit network interface cards. Additionally it has a PERC H700 controller card with 512MB of on-board cache. We will be attaching hypervisors to the dell server (block level storage), which will store virtual machines.

Our question is, using iSCSI, and a single gigabit connection between each hypervisor and the dell storage server, is it true that maximum theoretical transfer rate per hypervisor would be 1000 Megabits / 8 = 125 Megabytes a second? Or, am I completely wrong, and iSCSI does some sort of compression and is able to achieve higher I/O throughput rates.

125 Megabytes a second actually is somewhat slow seeing as we have 12 spindles and running RAID 10. What are some alternatives, besides fiber channel to remove the network bottleneck? We are aware of enabling jumbo frames, and will give that a try, anything else. What sort of performance should we expect using a single gigabit connection per hypervisor?

4 Answers 4

7

I'd say you'd be lucky to break 100MB/s. In theory, yes, you could transfer 1000Mbps, or 125MB/s, but between the various layers of overhead (Ethernet and IP headers, iSCSI itself, the fact that some time has to be spent between packets) you will never actually see that.

Also, don't forget that 125MB/s (or less) is what you'll see flying out of the NAS box; that has to be shared between all of the VM servers. Hence, don't expect to see that going to each VM server.

To speed things up, either go to 10Gbps networking (not cheap), or use Etherchannel/channel bonding/LACP/whatever your particular vendor likes to call it, and glue multiple 1Gbps links together to form a larger pipe. if that isn't an option (the 1RU servers I've seen have one expansion port), then you might want to consider an alternate protocol -- personally, I think ATA over Ethernet is a sadly neglected option if you're looking for a good SAN (as opposed to NAS) protocol.

Also, note that 7200RPM SATA drives really have poor performance, especially on random I/O, and, possibly more importantly, tend to have fairly annoying issues surrounding error handling (even the so-called "Enterprise" drives aren't what I'd call "adequate" in a high-performance SAN environment). I've managed a SAN that used these sort of drives, and quite frankly I'd spend the extra money on decent drives if I was going to do it again.

3
  • 1
    Instead of LACP, consider using MPIO with iSCSI as LACP may have scaling issues with 1 host and 1 SAN, depending on the configuration
    – tegbains
    Aug 16, 2011 at 6:37
  • I think we are going to look at ATA over Ethernet. Looks like openfiler supports it.
    – Justin
    Aug 16, 2011 at 7:00
  • 3
    @tegbains: That's a good point for the unwary to keep in mind: most forms of link aggregation have big problems reasonably balancing traffic when you have a single source or destination.
    – womble
    Aug 16, 2011 at 8:02
3

iSCSI has no compression, and the expected tranfer rate will be even lower when you take (Ethernet, IP and iSCSI) protocol overhead into account.

The easiest way to increase that is likely to put more NICs in both the storage server and the hypervisors and use port aggregation.

2
  • Unfortunately, we can't add another NIC to the hypervisors, they are 1U, and don't have room for expansion. So basically, whatever speed, we get from the gigabit connection, is it.
    – Justin
    Aug 16, 2011 at 6:28
  • Most 1U boxes have 1 or two PCIe slots for expansion. Are they taken? Also, you can get 4port NICs i.e. from Intel.
    – Sven
    Aug 16, 2011 at 6:40
2

Old question but here's the detail.

1000Mbit/s is the maximum bitrate. You might think that this therefore = 125MBytes/s.
No it doesn't. You see 1000Mbit/s is the physical bitrate at the link level. For a start there are protocol overheads. TCP + IP.

Just under 95% of the actual bandwidth of 1000Mbit/s is available after you had TCP/IP. Based on 1500 byte MTU with no VLAN & IPv4.

See: Protocol Overhead

Now, adding iSCSI. It appears as well that iSCSI add's a 48 byte header.
So with an MTU 1500, what's left is 1412 (1500-40-48) bytes. Therefore our throughput will drop to ~91% of maximum. ((1500-88)/(38+1500))

So without using jumbo frames expect to see an absolute maximum of 114MB/s. That assumes a perfect transfer with the whole frame being filled and no dropped packets.

In reality I'd expect you to drop below that especially if other users are hitting the target machine.

Notes:

  • iSCSI does not add any compression.
  • LACP may help alot.
  • Turning on Jumbo frames may help somewhat as it lowers frame/overhead compared to actual data.
2

LACP won't help. It benefits when multiple ESX hosts visit one iSCSI target but it won't increase bandwidth for a one on one situation. MPIO is the only way to increase bandwidth.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .