vSphere 5/Dell MD3000i multipathing configuration clarification

Question

Revisiting an older vSphere setup with two Dell 1950s, 4 NICs per host: esxi1 and esxi2 for simplicity sake.

SAN is a Dell MD3000i, two controllers, two NICs per controller: rc00 and rc01; rc10 and rc11

Only one LUN 0/virtual disk configured right now; RAID 10, 300GB SAS 15K, 6 spindles. Controllers/channels are as follows:

rc00: 192.168.130.101/24

rc01: 192.168.131.101/24

---

rc10: 192.168.130.102/24

rc11: 192.168.131.102/24

Switches (sw-1 and sw-2) are Dell PowerConnect 5424s; iSCSI "optimization" (QoS) is not enabled as there's no other traffic on the two switches. Jumbo Frames enabled, 9000 MTU, Flow Control on, MDIX auto.

Wanted to do some benchmarking while this setup is empty and I have some time on my hands.

Couldn't quite remember how to setup multipathing as it has been a while so, Googling around and reading a couple of older 4.1 white papers from Dell and vmware, I'm seeing really two ways of doing it:

one vSwitch with multiple VMKernel Ports and physical NICs:

rc00:192.168.130.101---sw-1----esxi1:vSwitch1:vmk1:eth1:192.168.130.11 rc01:192.168.131.101---sw-2----esxi1:vSwitch1:vmk2:eth2:192.168.131.11

... or two vSwitches with one VMKernel Port and one physical NIC:

rc00:192.168.130.101---sw-1----esxi1:vSwitch1:vmk1:eth1:192.168.130.11 rc01:192.168.131.101---sw-2----esxi1:vSwitch2:vmk1:eth2:192.168.131.11

Question #1: Are there any practical differences in performance or a reason to choose one over the other? Everything else look ok?

Question #2: I've actually staggered the VMKernel Ports across NIC controllers so that one of the VMKernel Port/physical NICs (eth1) is bound to one of the built-in Broadcom NICs, and the other (eth2) is bound to one of the Intel NICs.

I figured if one of the NICs/NIC controllers goes south, then there's still a path available through the second NIC/NIC controller. Wondering though if this will causing multipathing performance issues or general flakyness; didn't see anything out there to indicate one way or the other.

Perhaps I'm anticipating a failure that will likely never fail that "nicely" (i.e. if there's a NIC failure, chances are the host will just freak out anyways).

NOTE: the "one vSwitch, multiple VMKernel Ports" method actually seems to freak out the ESXi host. Takes an abnormally long time to reboot, some times the paths/LUNs are not showing active/active I/O or showing up at all, requiring a Rescan and/or and up/down of the VMKernel to get it to see the LUNs again. It looks odd for a configuration anyways as you're putting two different subnets on the same vSwitch/broadcast domain, and I believe that the vSwitches function as a layer 2 switch.

---

Benchmark #1: doesn't this seem kind of terrible?

Running ubuntu 10.04.2 LTS with "typical" settings (1 vCPU, 1024 MB RAM, 8 GB disk, defaults for filesystem, ext4 with LVM) and bonnie++:

gravyface@testubu:~$ bonnie++ -f -d /tmp
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
testubu          2G           96131  57 33783  16           98930  17 444.6  13
Latency                         623ms     645ms               111ms     503ms
Version  1.96       ------Sequential Create------ --------Random Create--------
testubu             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
          files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
             16 16509  79 +++++ +++ 25608  88 19044  86 +++++ +++ 25079  86
Latency             10289us    1398us    8288us     509us     442us   12159us

Take 2:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
testubu          2G           97240  54 32974  17           93371  17 420.6  14
Latency                         291ms    1421ms              1266ms     616ms
Version  1.96       ------Sequential Create------ --------Random Create--------
testubu             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
          files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
             16 14410  71 +++++ +++ 22082  86 18109  88 +++++ +++ 22054  88
Latency               108ms    1324us    2400us     814us      88us    4835us
1.96,1.96,testubu,1,1336168050,2G,,,,97240,54,32974,17,,,93371,17,420.6,14,16,,,,,14410,71, +++++,+++,22082,86,18109,88,+++++,+++,22054,88,,291ms,1421ms,,1266ms,616ms,108ms,1324us,2400us,814us,88us,4835us

Take 3: with --iops=3 set from esxcli

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
testubu          2G           115663  61 35594  18           103602  21 440.0  17
Latency                         285ms     571ms             52049us     477ms
Version  1.96       ------Sequential Create------ --------Random Create--------
testubu             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
          files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
             16 14206  73 +++++ +++ 22753  90 18424  91 +++++ +++ 22367  90
Latency               108ms    1951us    1827us    6200us     326us    6127us
1.96,1.96,testubu,1,1336168752,2G,,,,115663,61,35594,18,,,103602,21,440.0,17,16,,,,,14206,73,+++++,+++,22753,90,18424,91,+++++,+++,22367,90,,285ms,571ms,,52049us,477ms,108ms,1951us,1827us,6200us,326us,6127us

Answer 1

Q1: One vSwitch per vmkernel port is the usual way of doing it, but I'm not sure if anything gets jerky if you do it any other way. vSphere 5 has a pretty strict compliance test that you must pass in order to bind a adapter to the iSCSI initiator, and it could fail if you're using a single vSwitch. But these are just my thoughts, not actual facts :)

Q2: I also use different NIC's for each vmkernel, as I've seen NIC's go down before.. you really do not want to loose all connectivity against your storage.. but then again, the likelyhood for something like that to happen isn't exactly big. It's also quite common in FC enviroments to use dual single-port HBA's instead of single dual-port HBA's. Better safe than sorry?

Either way - you should not experience any performance issues since all modern NIC's have offloading built in. I would actually guess that you get better performance with dual NIC's, as you get different interrupts and a separate PCIe lane..