7

We are looking to setup a 3 node Hyper-V cluster with live migration and failover using:

  • 3 x Dell R710's with dual quad core Xeon & 128 GB RAM & 6 NICs in each
  • 1 x Dell MD 3220i SAN

We will be running this setup from a data center and so co-locating our kit.

Can anybody explain how we should setup the network connections to make the system redundant?

We have looked into this great article but are not sure how to get a 3 server setup correctly and reliably: http://faultbucket.ca/2011/01/hyper-v-failover-cluster-setup/.

I believe we need network connections for: live migration, heartbeat, management, hyper-v etc.

I assume as we are running it from a DC all IPs will have to be public IPs?

Are AD servers will be VMs. One on each Hyper-V server and setup not to be HA.

5
  • 1
    I don't want to sound snarky, but virtualisation clustering and high availability is very difficult to get right, it really might help to pay for an expert to come in and set it up and train you. I know Dell offer this service (it was about $17,000 if I remember correctly though). Feb 7, 2011 at 21:11
  • Thanks. We have asked Dell to quote for that but i'm not sure if they will setup the IP/Data Center bit? They are quoting £2,000.00 Feb 7, 2011 at 21:14
  • they will send you out a standard scope of works, and deviations from that are expensive. The scopes of work are pretty detailed. $17k included (for us) a PS-series array, VMWare, iSCSI setup and configuration as well as basic training and P2V migrations. We decided that we could spend that 17k training someone in-house to do it, so we didn't take their offer, so I don't know how it actually works in real life. Feb 8, 2011 at 6:49
  • "I assume as we are running it from a DC all IPs will have to be public IPs?" - not at all, use internal IPs whenever possible. Either buy VPN accounts from the DC or setup it yourself. I also agree that AT LEAST one AD should be outside of this configuration, or the Hyper-V servers should not be part of the AD as that might render you unable to logon in case of failure..
    – 3molo
    Feb 8, 2011 at 8:04
  • I'm a bit late to the party, only thing to really reitterate is that Hyper-V Clusters require AD to start, you need at least one AD server outside the cluster.
    – Chris S
    Mar 2, 2011 at 16:54

5 Answers 5

4

I've been there! Last year I set up a similar cluster (except my boxes were Fujitsu) using an iSCSI SAN for Hyper V.

Actually it's not that hard, but there are going to be snags along the way. If you are colocating, I would definitely be rehearsing your installation in a server rack on your own premises before moving it to the datacentre (I used a soundproofed server cabinet for this).

Oh, one other thing in preparation, you don't mention this, but one thing I wouldn't bother with is iSCSI boot which is offered on some iSCSI systems. It's a pain to set up and it doesn't always work with redundnancy. It's always better to have one or two physical boot disks on the nodes so that if you have a network configuration problem or iSCSI issue, you can still boot them up. I use small (40GB) solid state drives in each of my servers for boot disks.

You definitely need a separate AD DC. In fact I started with a 3 node cluster and then limited this to 2 nodes, plus an unclustered 'master node' which runs backups on DPM 2010 and a virtualised DC.

You mention 6 ports, this may be enough. But allow me to illustate my node configutation which has 10 ports:

  • You will always need 2 ports per node for the iSCSI network (each one belongs to a different subnet for MPIO redundancy and should be on separate NICs)
  • 1 port for heartbeat that doesn't have any other traffic on it
  • 1 port for live migration and transfers (this is the only one that you might want to upgrade to 10g or infiniband, but unless you are provisioning 10s of VMs per day it is not worth it)
  • 1 port for remote desktop access
  • 1 port for server management card
  • 1 set of teamed ports that constitutes the main access network*
  • 1 set of team ports that consitutes a DMZ network (optional)*

*It is often pointed out by detractors that Microsoft does not officially support port teaming (whereas VMWare does) but actually official word is that they don't discourage it, but simply feel that support is in the hands of the NIC vendors. I use Intel NICs that are of the ET generation, which have specific virtual network features and I find work very well with Hyper V. They actually allow you to split a team between switches so that if one of the switches fails you have consistent team access, a bit like MPIO but for virtual machines.

Hyper V is actually really resilient and good to use. I would approach your job in this sequence: 1) Set up the nodes individually, install iSCSI initiator, install MPIO, give your iSCSI ports, transport and heatbeat ports, management ports different subnet addresses.
2) Set up Hyper V and assign your chosen ports to your virtual network 3) Then run the cluster validation wizard and form the cluster.

You should always assign ports to your virtual network first, because this prevents them from being used by the cluster. This sounds counter-intuitive, but basically you are going to be keeping your virtual network agnostic to your cluster network. This still gives you redundancy, so don't worry. But to achieve this (and there is no other way) you are going to have to have either a separate set of switches for your cluster and hyper V (two each for redunancy) or you will need to set up VLANs on your switches. I do the latter (using untagged VLANS) and it works great.

Some other posts here have suggested that you use a consultant to do this work. If they are familiar with Hyper V, this may be a good idea. It doesn't give you the indepth knowledge that you'd otherwise acquire from DIY, but it will save you time. I had plenty of time last year, and I'm not embarassed to admit that it took me several months of work and figuring things out to get it all up and running.

Good luck!

1

Welcome in a world of pain. YOu make one mistkake that will ruin your experience in a momnent when you are in a world of pain for not thinking things through.

Are AD servers will be VMs. One on each Hyper-V server and setup not to be HA

THINK what you do there. Windows clustering needs AD to start, as the config is in AD. IF for anyy reason power fails in the data center, when it comes back up the clsuter will not start as there are no AD servers ready. YOu have to be 100% carefull with this.

I strongly suggest you keep an additional small machine (Atom based if it has t obe) as acting as controlling AD (i.e. important roles) with a separate USV. CLustering works nice, but having all AD controllers in VM's asks for trouble.

http://technet.microsoft.com/en-us/library/ff428137(WS.10).aspx

has the guide you need. On top of that, consider using a fast network backbone. No, not 10g... too slow, too expensive. Get a nice set of Infiniband cards, an Infiniband switch and be habppy with FAST transfers.

The SAN may also be too small in IOPS. If you talk 100% load on two machines (one in reserve) That is a LOT of VM's. You have to amke sure your IOPS budgets are in line for this. I have problems with a 6 10k Raptor Raid 10 on a single 64gb host at times - you run 4 times the memory budget, so I would expect 4-6 times th IOPS needs (and I have no databases there, those are separate). Make sure you know the SAN side is good enough.

2
  • How would one go a head and calculate the actual needed IOPS (in average and peak)?
    – 3molo
    Feb 8, 2011 at 8:07
  • Assuming. Seriously - depends TOTALLY On the load. I basically start with an expandable RAID array and make sure I have enough sapce to plug in discs (24 in my case). For a setup like yours I would start possibly with a 12-16 RAID 10 10k discs.
    – TomTom
    Feb 8, 2011 at 11:38
1

I'm reading about the same network configuration, and here's briefly what I think is correct. Please correct me if Im wrong.

  • 2 (etherchannel/hp trunked) for live migration
  • 2 (etherchannel/hp trunked) for normal network connectivity
  • 2 for SAN, no etherchannel as windows server can do MPIO

Use two switches. If not possible to to stack, create an etherchannel / hp trunk between them and use (at least) as many ports for this as you have active to your SAN. ports for this.

  • On the ports for the SAN, and to the SAN nics on the servers, turn on Jumbo frames and flow control. This should also be set on the inter-switch links.
  • Inter switch ports should be etherchannels/hp trunk ports, I think both Jumbo frames and flow control enabled
  • Live migration links etherchanneled (hp trunks), jumbo frames and flow control.

Turn off unicast storm control on the switches. If possible, use other switches for network connectivity. Disable Spanning Tree-protocol.

Regarding the 3220i SAN, I think it has 4 Gbit ports, two on each controller. I think it's a active passive, and if connect one each of every controllers NIC to one switch each.

Forgot to mention that you should separate live migration and SAN on different VLANs. Avoid default VLAN (1). Use good switches, the buffers (or something) have a rather big impact on the performance (so I read).

0

have you looked at the following

http://technet.microsoft.com/en-us/library/ff428137(WS.10).aspx

I find it quite hard to screw up hyper-v cluster so go for it.

0

While Windows clustering does require AD to start, you can make it work very well in 2-node totally virtualized environments.

I would make these recommendations that have served me well for both our internal network and our customer networks: - Reserve enough disk space and I/O on each node of your cluster to run a local DC. That way, the cluster service never has look far to find something to query against. - Set the cluster service to "Automatic (Delayed Start)." That way, Hyper-V has a little extra time to start before the cluster service.

1
  • 1
    While this probably does work for small installations; generally Hyper-V Clusters will be of sufficient size to warrant the Hyper-V hosts being dedicated and standardized to their task, and an external Active Directory box. This is very much like SBS boxes having AD, Exchange, and SQL all on the same OS; it might work for very small installs, but it's decidedly not best practices.
    – Chris S
    Mar 22, 2012 at 2:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .