Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a pile of HP SAN and server boxes behind my desk.

Long story short, I'll have 2 vSphere hosts, each with 12 pNICs, and a single Procurve 2910al switch (actually two but linked) dedicated for iSCSI/vMotion traffic, and the iSCSI SAN (P4000).

Some NIC's will be allocated to my production LAN as VM NICs and so I can access the vCentre server, and the iSCSI traffic and anything best kept of the production LAN will go on the 2910al.

I want to present iSCSI LUN's to some of the guests (Exchange/SQL/maybe the file server) using the Windows iSCSI initiator so I can use the SAN integrated VSS snapshots.

I also ideally want to be able to manage the SAN from the production network, so I guess I'd use the routing function on the switch for that?

I'd appreciate suggestions on the optimal way to configure the switch/VLAN layout.

share|improve this question
up vote 4 down vote accepted

From the vSphere perspective you want redundancy on the Service console\Management network. Ideally two separate pnics connecting into two independent switches. So two nics there on an isolated management VLAN.

Vmotion\Fault tolerance (if you are using the latter) again at least two pnics connected to two independent switches. So two nics there on a separate VLAN on the procurve 2910(s).

For iSCSI that will be presented to the vSphere environment you again want at least two and in this case it is very, very important that they connect to independent switches. From a cSphere perspective the recommendation is to configure as many separate vSwitches, with a single VMKernel port on each and a single pnic. Each of these vmKernel ports should have vMotion\FT traffic disabled. All iSCSI vmkernel ports then need to be bound to the iSCSI stack to enable failover, and multipathing if that is an option for your array. It is possible to get load balanced native multipathing without needing Enterprise Plus provided your vendor has a Multipathing Extension Module (MEM), if they provide a Path Selection Provider (PSP) then Enterprise Plus is required. I'm not sure how the LeftHand handles this, worst case is that you get failover but no actual load balancing. On the switch front keep the iSCSI connections on their own VLAN on the Procurves.

For native iSCSI presented to the VM's you want the same sort of resilience, with at least two pnics connected to separate physical switches. Ideally you should repeat the pattern used for the vSphere iSCSI switches - one pnic per vSwitch and one VM port group per vSwitch. That enables the internal multipathing components within the VM's to make sensible path management decisions and gives them some visibility onto the connection state of the path. If load balancing isn't important for you in the VM's then simple teaming and a single vSwitch will do but given your plans that doesn't seem optimal. Again these connect to the Procurves.

I'd very strongly recommend that your two Procurves are not configured as a single logical unit (using stacking) if at all possible. Lots of inter switch links configured as a single LAG is better, 10Gig if possible, from a manageability and safety perspective. Consider how you will upgrade firmware on these switches or carry out other maintenance down the line.

In any case that uses up 8 of your 12 pnics on each vSphere host leaving you with four for production traffic which seems reasonable enough.

You can then manage the iSCSI environment from VM's that have vnics connected to the iSCSI VM Port groups if you like, otherwise you will need to provide some connectivity between the procurves and your production environment. Definitely handle that at layer 3 - you want to keep the iSCSI environment as free from extraneous traffic as possible at layer2.

share|improve this answer
+1. Great point about not stacking the switches. – Jed Daniels Dec 3 '10 at 18:24
Thanks Helvik, very comprehensive reply. Lefthand suggests a single vSwitch for iSCSI with multiple VMKernel's on each - you're suggesting multiple vSwitch's? I also wondered if there's any mileage in using port groups, which AIUI let's me use the same pNIC's for multiple roles - for example I want redundancy on the SC but I don't need 2gbps of bandwidth, I'm wondering if I could share the pNICs with another role (vMotion?) and use port groups to VLAN the traffic? The 2910's will be connected by a 10gbps link, think stretch cluster, they are SPOF but that's how it is for now. – Hutch Dec 3 '10 at 18:41
You can do it with a single vswitch that has multiple vmkernel ports and then use the nic failover order settings on the vmkernel ports to ensure that there is a 1::1 mapping between the VMkernel ports and pnics but that's not really significantly different. The key thing is that the VMkernel ports have to map to one, and only one, pnic with no teaming\standby for iSCSI multipathing to work on vSphere 4 and above. You can share the pnic with other functions (by adding a VM port group on the same vSwitch) if you are happy that that wont impact performance. – Helvick Dec 3 '10 at 23:46
The relevant document from VMware is the vSphere iSCSI configuration guide ( ) and it applies to Lefthand. This article discusses some more specifics for Lefthand in terms of enabling multipathing - – Helvick Dec 3 '10 at 23:51
Thanks Helvik, those do tie in with what I was expecting to be doing with the iSCSI side of things. Fortunately we have quite a bit of overlap in how long we have to replace this kit so I have opportunity to try a few things. VLAN's will be interesting as I think 4gbps should be plenty to support Management, vMotion, and VM traffic in our environment, so it's whether I can get the switches and port groups setup correctly to segregate the vMotion traffic on a separate VLAN that never sees our production LAN. – Hutch Dec 4 '10 at 10:00

I'm guessing that no one is really going to be able to provide specifics without more information regarding what edition of vSphere you've got (only Enterprise and Enterprise Plus support some of the advanced features like storage multipathing), in addition to what your requirements are for BW usage and fault tolerance. With that said, when planning for this stuff, I always break the NICs down into the following four categories, then determine what my requirements are for each, followed by what can actually be accomplished:

  1. Management
  2. vMotion
  3. Storage
  4. Distributed vSwitch (assuming Enterprise Plus)

Do you need redundancy for each? Do you need more than 1G for each? Do you have enough NICs to provide redundancy? (In this case, you do, IMHO.) Once you make these decisions then planning what to actually plug in where should be a piece of cake (assuming that you are going to maintain separation between the four networks above, which I recommend).

So it looks like you'll need at least

  1. 2 x Management
  2. 2 x vMotion
  3. 4 x Storage - Hosts (2Gbps x 2 for redundancy)
  4. 4 x Storage - Guests (2Gbps x 2 for redundancy)

Good thing you got 12 NICs in each host, cause you are already at the limit!

Assuming the switches behave as a single unit, you'll need a VLAN for management, one for vMotion, and one for Storage, and you'll connect everything up so that each group has equal connections to the two physical switches.

share|improve this answer
I'm fairly certain (as proven by one of my vCenter clusters) that multipath is fully-supported with all versions of ESXi (even the free version). – EEAA Dec 3 '10 at 17:27
Thanks Jed, and sorry, vSphere Standard. I figure have enough NICs to make everything redundant, and for the iSCSI BW I envisage 2gbps per host for the vSphere iSCSi initiator and 2gb per host to the various guest initiators to be sufficient. The 2910's are solely for iSCSI and any vSphere stuff that's best kept off the production LAN. – Hutch Dec 3 '10 at 17:31
@ErikA According to vStorage APIs for Array Integration, and vStorage APIs for Multipathing, are only available in Enterprise and higher; Storage vMotion, Storage I/O Control, Network I/O Control, Distributed Switch, and Host Profiles are only available in Enterprise Plus. So I guess you are right, multipathing is supported, but not the APIs, and not Storage vMotion and Storage I/O control, which may impact the planning and deployment in this situation. – Jed Daniels Dec 3 '10 at 17:52
Standard will be sufficient for our relatively simple needs. We'll be doing multi-pathing using the vSphere S/W iSCSI initiator and any other multi-pathing will be done from within the guest using either the Windows iSCSI initiator or the DSM from Lefthand. – Hutch Dec 3 '10 at 17:57
@Jed - yep they reserve all the tasty bits for the very highly-priced versions. :( – EEAA Dec 3 '10 at 18:02

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.