5

So this is my new "project" which consists of tearing down our legacy network and re-doing it correctly. As of now it is a mess. Production is a hogpoge of legacy servers that should have been put to pasture two years ago. Within this time period I got my VCP, which is good but I do not know all of this by heart. This is my first network / server revamp. As well as my first non-lab vSphere work, actually I have never logged into a SAN up until a few weeks ago when we received our PS4000 At this point am breaking this project into design phases so the first one up is the VLAN, Switching layout. I have attached my spread sheet with vlan id's, subnets, use and switch ports. I have read allot of best practices but, it is starting to just melt together. If i could get any advice on this I would very much appreciate and, please feel free to rip it apart.

VLAN Assignments Rev.1

http://dl.dropbox.com/u/6674806/Copy%20of%20Production%20VLAN%20Port%20Assignments-1.xlsx

enter image description here

Kind Regards,

Brandon

VLAN Assignments Rev. 2

Updated layout

Uploaded a updated version per advice.

3
  • Whatever it is that you've linked to is broken. I don't know if's the URL shortner or whatever the URL points to, but it never loads anything (I don't know why you've chosen to use a URL shortner anyway, this isn't twitter) Jun 7, 2011 at 23:38
  • I had to move it to a different folder but, it has the full link now.
    – BrandonB
    Jun 7, 2011 at 23:46
  • Might be going a bit overboard on VLANing; my two bits.
    – gravyface
    Jun 8, 2011 at 1:35

3 Answers 3

4

You have done this absolutely correctly! It appears to me also that you're using Juniper switching based on the format of your interfaces.

Recommendations:

Make sure you set MTU of 9216 on the Juniper switch for any port carrying iSCSI. Equallogic likes jumbo frames. You should set ALL the links from the switch to your ESXi systems as trunks, carrying the appropriate VLANs, this gives you flexibility.

Finally, install the Dell flavored version of ESXi so it has the open manage and systems health plugins, and afterwards install the Dell multi-pathing plug-in for ESXi and let it create your iSCSI vswitch for you, this makes sure that the vswitch, and in turn each vmnick and vmkernel port all are set correctly for jumbo frames.

Remember also how vmware blocksizes affect the datastores on your SAN, typically unless there is a requirement we create a datastore with a 1MB block size, but this limits you to a VMDK of like 512GB or something like that I can't remember, a 2MB block size will let you do 2TB VM's, and so on. (I can't remember the figures).

5
  • also, when you create the EX4200 juniper stack, under the virtual chassis section you should add the no-split-detection statement so you don't run into a split-brain scenario since you only have two switches. Jun 8, 2011 at 0:56
  • I used to create 3 vswitches, one for mgmt and vmotion, one for vmnetworking, and one for iSCSI, after many many installs i'm down to just two, one for everything BUT iSCSI, and then the iSCSI dedicated vswitch. Jun 8, 2011 at 0:59
  • also, do not try and get the hardware iSCSI working, you'll want to rip your eyeballs, out, use the Dell plug-in and let it use the software initiator :) Jun 8, 2011 at 1:01
  • 1MB blocks = 256GB max vmdk, 2MB = 512GB, 4MB = 1TB, 8MB = 2TB. I'd lean toward tearing your eyes out on getting the hardware iSCSI working, if you've put the extra cash on getting a NIC with hardware iSCSI offloading. Jun 8, 2011 at 1:21
  • Thanks Shane... another guy usually does the storage for me. That being said, I've had nothing but trouble with the iSCSI offloading and the Dell MEM plugin makes it so easy to round-robin load balance that I kinda gave up on it. Jun 8, 2011 at 2:12
4

@SpacemanSpiff has put some good recommendations out there, a couple things to add:

  1. I don't see any notes on which interface the VMware HA communication will be using. By default it'll probably grab the management interface, which should be fine; it's not a lot of traffic. But make sure you're aware of which interface it's using; if, for instance, you added a host on a different vlan for management but the same VMotion and FT vlans, then the HA cluster wouldn't accept that host.

  2. Not sure what kind of physical NICs you've got, but I'd recommend making sure a given vSwitch has uplinks using multiple physical NICs, so a card failure can't bring you down. So, for instance if you've got 2x 4-port NICs, then instead of doing management with vmnic1 and vmnic2, do it on 1 and 5.

1
  • From the spreadsheet it looks like he's got a boatload of NICs.
    – gravyface
    Jun 8, 2011 at 1:36
3

The overall approach is sound. Some comments:

  • You may have too many adapters and at the same time too little bandwidth. Specifically, there's no reason to have 2 pNICs for management and 2 pNICs for vMotion. Use 2 pNICs for both, but configure one to be active for management and standby for vMotion, and the other active for vMotion and standby for management. As for bandwidth, FT over gigabit may not work. You need to calculate the needs for FT using the formulas in http://www.vmware.com/files/pdf/perf-vsphere-fault_tolerance.pdf.
  • For iSCSI, you must use Round Robin or EQL MEM extension, not just any multipathing.
  • Use iSCSI-enabled Broadcom NICs and configure hardware offload per http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf. Keep in mind that Broadcoms are not real expensive iSCSI HBAs, and they don't support jumbo frames. Still, you'll have better performance than with pure software iSCSI and jumbo frames.
  • You list various VMs that will use different VLANs, but you only specify a single VLAN for your "data" (VM network, I assume?) pNICs. Instead, they should be configured as trunks, with multiple portgroups on the VM side.
  • Look at your budget numbers and decide if you want to bump up the license level to Enterprise. That will give you DRS, Storage vMotion, and VAAI. It is also a prerequisite for EQL MEM. Since you're building a three server cluster, your best purchase is one of the acceleration kits, and the difference between Adv and Mid (w/ Ent. ESXi) is ~$5K.
  • Buy Dell Management Plug-in for VMware. This will be your favorite addon after MEM.
  • Don't skimp on memory, especially if your VMs are mostly Windows. Pack those PowerEdges with as much RAM as your budget allows.

I'm sure I could think of more, but this should be helpful too.

6
  • I almost made the top point as well, we typically only use two links total for management and vmotion, in the fashion described above. Jun 8, 2011 at 2:15
  • Oh and DMZ will be "DATA". One more question FT best practices says that I should run it on its vlan and then add management and vmotion combined ? I have 12 pNIC's per box so I thought it would be great to have a physical nic per each service.
    – BrandonB
    Jun 8, 2011 at 4:35
  • 1
    Separate FT to its own pNICs and VLAN. Combine vMotion and management in the same vSwitch with two pNICs, assigning the two vmknics to different pNICs. This way the only case when vMotion and management will share a network path is if one of those two pNICs fails. And your DATA is not just DMZ. It's everything - including management VLAN for VMs (don't put VM traffic where your management vmknic is). So, the two DATA pNICs should be trunks, not access ports. Jun 8, 2011 at 6:08
  • I have 1x "Embedded Broadcom, GB Ethernet NICS with TOE and ISCSI Offload Enabled" ( 4 ports ) and 2x Intel Gigabit ET Quad Port NICPCIe-4 ( 8 ports ). I assumed I would just use the broadcom's for iscsi any thoughts as far as performance / experiance on these two controllers ? Thank you for looking this over.
    – BrandonB
    Jun 8, 2011 at 16:10
  • ynguldyn the reason I have to interfaces for management is for redundancy but, if thats not needed i can use those else were. Any thoughts on my pNIC allocation we will have 20 VM's running is two pNIC's sufficient ?
    – BrandonB
    Jun 8, 2011 at 18:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .