Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

New SAN for me, never used before: it's an IBM DS3512, dual controller with a quad 1GbE NIC per controller that a client bought and needs help setting up.

Hosts (x2) have 8 pNICs and while I usually reserve 2 pNICs for iSCSI per host (and 2 for VM, 2 for management, 2 for vMotion, staggered across adapters), these extra ports on the SAN have me wondering if storage I/O would be significantly improved with 2 additional NICs per host, or if the limitations of the vmkernel/initiator would prevent the additional multipaths from ever being realized.

I'm not seeing alot of 4 pNIC iSCSI implementations per host; 2 is the de facto standard from what I've read/seen online. I could and probably will do some I/O testing, but just wondering if there's a "wall" that someone else has discovered long ago (i.e. before 10GbE) that makes a 4 NIC iSCSI per host setup somewhat pointless.

Just to clarify: I'm not looking for a how-to, but an explanation (link to paper, VMWare recommendation, benchmark, etc.) as to why 2-NIC configurations are the norm vs. 4-NIC iSCSI configurations. i.e. storage vendor limitations, VMKernel/initiator limitations, etc.

share|improve this question
    
How many hosts? It could (and usually does) make a lot of sense for the storage array to have much more bandwidth than each single host. –  Massimo Dec 9 '12 at 16:46
    
Just two hosts, and yes, very good point. –  gravyface Dec 9 '12 at 16:47

2 Answers 2

If I were in your position, I'd assess whether the I/O needs dictate more than the bandwidth and path selection offered by the two physical NICS and 1GbE iSCSI. Honestly, I use 10GbE more than anything nowadays, but with proper MPIO configuration, there's no harm in adding additional iSCSI ports.

What's your VMWare license level? If you're not using DRS, for instance, some of this may be moot.

As for making your multiple-path iSCSI more effective, you'll want to change the path selection to Round-Robin and lower the number of I/O operations before switching paths from the default of 1000 to 1... That's what I use for HP and other SAN solutions.

The IBM DS3512 specific implementation guide is here.

share|improve this answer
    
Thanks ewwhite, but I'm looking for an explanation as to why 4-NIC iSCSI configurations are seemingly rare (i.e. limitations in the initiator, kernel, storage vendors, etc.). –  gravyface Dec 9 '12 at 17:11
    
@gravyface Because it's typically not necessary. If I/O is an issue, more paths are used or people go to higher-bandwidth connections. To be honest, most VMWare installations I see typically neglect to make the path selection (round-robin) changes I mentioned in my answer, so a good amount of performance is usually left on the table. –  ewwhite Dec 9 '12 at 17:16
    
e.g. if you're not making the path changes, adding more pNICS to your iSCSI storage links won't benefit you. –  ewwhite Dec 9 '12 at 17:17
    
...and 10GbE proliferation is probably another likely reason. –  gravyface Dec 9 '12 at 17:23

As long as you stick to the "each path on a different subent and network adapter" rule, this should work fine.

  • Define 4 IP subnets (ideally each one on its own VLAN or physical switch).
  • For each Storage Controller:
    • Connect port 1 to subnet 1, port 2 to subnet 2, etc.
    • Configure each port to have an IP address in the correct IP subnet
  • For each host:
    • Define which ports to use for iSCSI.
    • Connect port 1 to subnet 1, port 2 to subnet 2, etc.
    • Create 4 vSwitches
    • Bind each vSwitch to one of the physical ports
    • Create a VMkernel port for each vSwitch
    • Configure the VMkernel port to have an IP address in the correct IP subnet

Then make your iSCSI connections.

share|improve this answer
    
Thanks Massimo, but not really looking for a howto, just curious to know if the reason why I'm not seeing alot of 4-NIC iSCSI configurations is because the additional paths were never utilized (i.e. not needed, limitations in the initiator, kernel, etc.). –  gravyface Dec 9 '12 at 17:00
    
There are no actual limitations, you can safely use this setup if you have NICs to spare. But what will be actually made of it depends on your workload and your storage array capabilities. At worst, you could make use of a single path and have the other ones only provide redundacy. –  Massimo Dec 9 '12 at 17:02
    
I'll probably stick with two as there's only 8 NICs in the host and redundancy/separation is more important than potential storage I/O performance improvements, but just wondering why I'm not seeing 4-NIC iSCSI implementations out there. –  gravyface Dec 9 '12 at 17:06
    
You don't actually need different subnets for multipath in vSphere. –  MDMarra Dec 9 '12 at 17:51
    
@MDMarra would you not then require stacked/LAGG'ed switches in that case to ensure that all interfaces are on the same broadcast domain? –  gravyface Dec 9 '12 at 18:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.